Kind: captions
Language: en
the following is a conversation with
Marcus hunter senior research scientists
the google deepmind throughout his
career of research including with
Juergen Smith Huber and Shayne leg he
has proposed a lot of interesting ideas
in and around the field of artificial
general intelligence including the
development of IHC spelled a ixi model
which is a mathematical approach to AGI
that incorporates ideas of Kolmogorov
complexity solomonoff induction and
reinforcement learning in 2006
Marcus launched the 50,000 euro hütter
prize for lossless compression of human
knowledge the idea behind this prize is
that the ability to compress well is
closely related to intelligence this to
me is a profound idea specifically if
you can compress the first 100 megabytes
or 1 gigabyte of Wikipedia better than
your predecessors your compressor likely
has to also be smarter the intention of
this prize is to encourage the
development of intelligent compressors
as a path to AGI in conjunction with
this podcast release just a few days ago
Markus announced the 10x increase in
several aspects of the surprise
including the money to 500,000 euros the
better your compressor works relative to
the previous winners the higher fraction
of that prize money is awarded to you
you can learn more about it if you
Google simply Qatar prize I have a big
fan of benchmarks for developing AI
systems and the harder prize may indeed
be one that will spark some good ideas
for approaches that will make progress
on the path of developing a GI systems
this is the artificial intelligence
podcast if you enjoy it subscribe on
YouTube give it five stars an Apple
podcast supported on patreon or simply
connect with me on Twitter at lex
Friedman spelled Fri D M am as usual
I'll do one or two minutes of ads now
and never any ads in the middle that can
break the flow of the conversation I
hope that works for you and doesn't hurt
the listening experience
this show is presented by cash app the
number one finance app in the App Store
when you get it
use collects podcast cash app lets you
send money to friends buy Bitcoin and
invest in the stock market with as
little as one dollar brokerage services
that provided by cash up investing a
subsidiary of square and member s IPC
since cash app allows you to send and
receive money digitally peer-to-peer and
security in all digital transactions
very important let me mention the PCI
data security standard that cash app is
compliant with big fan of standards for
safety and security PCI DSS is a good
example of that or a bunch of
competitors got together and agreed that
there needs to be a global standard
around the security of transactions now
we just need to do the same for
autonomous vehicles and AI systems in
general so again if you get cash out
from the App Store or Google Play and
use the code Lex
Podcast you'll get ten dollars and cash
app will also donate ten dollars the
first one of my favorite organizations
that is helping to advance robotics and
STEM education for young people around
the world and now here's my conversation
with Markus cutter
as a computer or maybe an information
processing system let's go with a big
question first okay I with a big
question first yeah I think it's very
interesting hypothesis or idea and I
have a background in physics so I know a
little bit about physical theories the
standard model of particle physics and
general relativity theory and they are
amazing and describe virtually
everything in the universe and they're
all in a sense computable theories I
mean they're very hard to compute and
you know it's very elegant simple
theories which describe virtually
everything in the universe so there's a
strong indication that somehow the
universe is computable but it's a
plausible hypothesis so what what do you
think just like you said general
relativity quantum field theory what do
you think that the laws of physics are
so nice and beautiful and simple and
compressible do you think our universe
was designed is naturally this way are
we just focusing on the parts that are
especially compressible our human minds
just enjoy something about that
simplicity and in fact there's other
things that are not so compressible no I
strongly believe and I'm pretty
convinced that the universe is
inherently beautiful elegant and simple
and described by these equations and
we're not just picking that I mean if
the versatile phenomena which cannot be
need to describe scientists would try
that right and you know there's biology
which is more messy but we understand
that it's an emergent phenomena and you
know it's complex systems but they still
follow the same rules right of quantum
electrodynamics and all of chemistry
follows that and we know that I mean we
cannot compute everything because we
have limited computational resources now
I think it's not a bias of the humans
but it's objectively simple I mean of
course you never know you know maybe
there's some corners very far out in the
universe or super super tiny below the
nucleus of atoms or well parallel
universes where which are not nice and
simple but there's no evidence for that
and you should apply Occam's razor and
you know just the simple story
consistent with but also it's a little
bit
for friendship so maybe a quick pause
what is Occam's razor so or comes razor
says that you should not multiply
entities beyond necessity which sort of
if you translate it to proper English
means and and you know in a scientific
context means that if you have two
series or hypotheses or models which
equally well describe the phenomenon
your study or the data you should choose
the more simple one so that's just the
principle you're sort of that's not like
a provable law perhaps perhaps we'll
kind of discuss it and think about it
but what's the intuition of why the
simpler answer is the one that is likely
to be more correct descriptor of
whatever we're talking about
I believe that Occam's razor is probably
the most important principle in science
I mean of course we logically Duck
shouldn't be do experimental design but
science is about finding understanding
the world finding models of the world
and we can come up with crazy complex
models which you know explain everything
but predict nothing but the simple model
seem to have predictive power and it's a
valid question why yeah and the two
answers to that you can just accept it
that is the principle of science and we
use this principle and it seems to be
successful we don't know why but it just
happens to be or you can try you know
find another principle which explains or
comes razor and if we start with the
assumption that the world is governed by
simple rules then there's a bias toward
simplicity and pliant Occam's razor is
the mechanism to finding these rules and
actually in a more quantitative sense
and we come back to that later in terms
of some Roman attraction you can
rigorously prove that usually assume
that the world is simple then Occam's
razor is the best you can do in a
certain sense so I apologize for the
romanticized question but why do you
think outside of its effectiveness why
do we do you think we find simplicity so
appealing as human beings well
just why does e equals mc-squared seems
so beautiful to us humans I guess mostly
in general many things can be explained
by an evolutionary argument and you know
there's some artifacts and humans which
you know are just artifacts and not an
evolutionary necessary but there's this
beauty and simplicity it's I believe at
least the core is about like science
finding regularities in the world
understanding the world which is
necessary for survival right you know if
I look at a bush right and I just seen
Norris and there is a tiger right and
eats me then I'm dead but if I try to
find a pattern and we know that humans
are prone to find more patterns in data
than they are you know like the you know
Mars face and all these things but these
buyers towards finding patterns even if
they are not but I mean its best of
course if they are yeah helps us for
survival yeah that's fascinating I
haven't thought really about this I
thought I just loved science but they're
indeed from in terms of just for
survival purposes there is an
evolutionary argument for why why we
find the work of Einstein is so
beautiful maybe a quick small tangent
could you describe what's Solomonov
induction is yeah so that's a theory
which I claim and Riesling enough sort
of claimed you know a long time ago that
this solves the big philosophical
problem of induction and I believe the
claim is essentially true and what it
does is the following so okay for the
picky listener induction can be
interpreted narrowly and wildly narrow
means inferring models from data and
widely means also then using these
models for doing predictions or
predictions also part of of the
induction so I'm little sloppy sort of
as a terminology and maybe that comes
from ray solomonoff you know being
sloppy maybe
saying it we can't complain anymore
so let me explain a little bit this
theory yeah in simple terms so assume we
have a data sequence make it very simple
the simplest one say 1 1 1 1 1 and you
see if 100 ones yeah what do you think
comes next the natural order I repeat up
a little bit the natural answer is of
course you know 1 ok and questions why
ok well we see a pattern there yeah ok
there's a 1 and we repeat it and why
should it suddenly after a hundred ones
be different so what we're looking for
is simple explanations or models for the
data we have and now the question is a
model has to be presented in a certain
language in which language to be used in
science we want formal languages and we
can use mathematics or we can use
programs on a computer so abstract me on
a Turing machine for instance or can be
a general-purpose computer so and they
of course lots of models of you can say
maybe it's a hundred ones and then 100
zeros and a hundred ones that's a model
right but there are simpler models
there's a model print one loop and it
also explains the data and if you push
the to the extreme you are looking for
the shortest program which if you run
this program reproduces the data you
have it will not stop it will continue
naturally and this you take for your
prediction and on the sequence of ones
it's very plausible right at the print
one loop it's the shortest program we
can give some more complex examples like
1 2 3 4 5 what comes next the short
program is again you know counter and so
that is roughly speaking house a lot of
interaction works the extra twist is
that it can also deal with noisy data so
if you have for instance a coin flip say
a biased coin which comes up head with
60% probability then it will predict if
you learn and figure this out and after
a while it predict or the next coin flip
will be head with probability 60% so
it's the stochastic version of that but
the goal is the dream is always the
search for the short program yes yeah
well in solomonov induction precisely
what you do is so you combine so looking
for the shortest program is like
applying AAPIs race
like looking for the simplest theory
there's also a pakoras principle which
says if you have multiple hypotheses
which equally well describe you data
don't discard any of them keep all of
them around you never know and you can
put it together and say ok have a
buyer's to her simplicity but I don't
rule out the larger models and
technically what we do is we weigh the
shorter models higher and the longer
models lower and you use a Bayesian
techniques you have a prior and which is
precisely 2 to the minus the complexity
of the program and you weigh all this
hypotheses and take this mixture and
then you get also this plasticity in
yeah like many of your ideas that's just
a beautiful idea of weighing based on
the simplicity of the program I love
that that that seems to me may be a very
human central concept seems to be a very
appealing way of discovering good
programs in this world you've used the
term compression quite a bit I think
it's a beautiful idea sort of we just
talked about simplicity and maybe
science or just all of our intellectual
pursuits is basically the attempt to
compress the complexity all around us
into something simple so what does this
word mean to you
compression I essentially have already
explained it so it compression means for
me finding short programs for the data
or the phenomena at hand you could
interpret it more widely as you know
finding simple theories which can be
mathematical theory so maybe even
informal you know like you know just
inverts compression means finding short
descriptions explanations programs
little data do you see science as a kind
of our human attempt at compression so
we're speaking more generally because
when you say programs kind of zooming in
a particular sort of almost like
computer science artificial intelligence
focus but do you see all of human
endeavor as a kind of compression well
at least all of science ICSI and evolve
compression at all of humanity maybe and
well they are so
other aspects of science like
experimental design right I mean we we
create experiments specifically to get
extra knowledge and this is that isn't
part of the decision-making process but
once we have the data to understand the
data is essentially compression so I
don't see any difference between
contrast compression understanding and
prediction so we're jumping around
topics a little bit but returning back
the simplicity a fascinating concept of
komagawa of complexity so in your sense
the most objects in our mathematical
universe have high komagawa of
complexity and maybe what is first of
all what is coma graph complexity ok
Kolmogorov complexity is a notion of
simplicity or complexity and it takes
the compression view to the extreme so I
explained before that if you have some
data sequence just think about a file on
a computer and best sort of you know
just a string of bits and if you and we
have data compresses likely compress big
files in terms a sip files with certain
compressors and you can also put
yourself extracting archives that means
as an executable if you run it it
reproduces the original file without
needing an extra decompressor it's just
a decompressor plus the archive together
in one and now there are better and
worse compressors and you can ask what
is the ultimate compressor so what is
the shortest possible self-extracting
archives you could produce for a certain
data set yeah which reproduces the data
set and the length of this is called the
Kolmogorov complexity and arguably that
is the information content in the data
set I mean if the data set is very
redundant or very boring you can
compress it very well so the information
content should be low and you know it is
low according to this difference this is
the length of the shortest program that
summarizes the data yes yeah and what's
your sense of our sort of universe when
we think about the different the
different objects in our universe that
we each are concepts or whatever the
at every level do they have higher or
local girl complexity so what's the hope
do we have a lot of hope and be able to
summarize much of our world that's a
tricky and difficult question so as I
said before I believe that the whole
universe based on the evidence we have
is very simple so it has a very short
description the whole sorry did you
would you linger on that the whole
universe what does I mean do you mean at
the very basic fundamental level in
order to create the universe yes yeah so
you need a very short program when you
run it to get the thing going you get
the thing going and then it will
reproduce our universe and there's a
problem with noise we can come back to
the later possibly noise a problem or a
fear is it a bug or a feature I would
say it makes our life as a scientist
really really much harder I didn't think
about without noise we wouldn't need all
of the statistics but that maybe we
wouldn't feel like there's a free will
maybe we need that for the ethics this
is an illusion that Norris can give you
freezing that way it's a feature but
also if you don't have noise you have
chaotic phenomena which are effectively
like noise so we can't you know get away
with statistics even then I mean think
about rolling a dice and you know forget
about quantum mechanics and you know
exactly how you you throw it but I mean
it's still so hard to compute a
trajectory that effectively it is best
to model it you know as you know coming
out this a number this probability 1
over 6 but from from this set of
philosophical como go of complexity
perspective if we didn't have noise then
arguably you could describe the whole
universe as well as standard model plus
general relativity I mean we don't have
a theory of everything yet but sort of
assuming we are close to it or have it
here plus the initial conditions which
may hopefully be simple and then you
just run it and then you would reproduce
the universe but that's all by noise or
by chaotic systems or by initial
conditions which you know may be complex
so now if we don't
the whole universe but just a subset you
know just take planet Earth planet Earth
cannot be compressed you know into a
couple of equations this is a hugely
complex just so interesting so when you
look at the window like the whole thing
might be simple when you just take a
small window then it may become complex
and that may be counterintuitive but
there's a very nice analogy the the book
the library of all books so imagine you
have a normal library with interesting
books and you go there great lots of
information and you quite complex yeah
so now I create a library which contains
all possible books say of 500 pages so
the first book just has a aaaa over all
the pages the next book aaaa and ends
with P and so on I create this library
of all books I can write a super short
program which creates this library so
this library which has all books has
zero information content and you take a
subset of this library and suddenly have
a lot of information in there so that's
fascinating I think one of the most
beautiful object mathematical objects
that at least today seems to be under
study or under talked about is cellular
automata what lessons do you draw from
sort of the game of life for cellular
automata where you start with the simple
rules just like you're describing with
the universe and somehow complexity
emerges do you feel like you have an
intuitive grasp on the behavior the
fascinating behavior of such systems
where some like you said some chaotic
behavior it could happen some complexity
could emerge some it could die out and
some very rigid structures you have a
sense about cellular automata that
somehow transfers maybe to the bigger
questions of our universe is a cellular
automata and especially the Conway's
Game of Life is really great because
this rule are so simple you can explain
it to every child and mean by hand you
can simulate a little bit and you see
these beautiful patterns emerge and
people have proven you know that is even
Turing complete you cannot just use a
computer to simulate game of life but
you can also use game of life to
simulate any computer that is truly
amazing
and it's it's the prime example probably
to demonstrate that very
simple rules can lead to very rich
phenomena and people you know sometimes
you know how can how is chemistry and
biology is so rich I mean this can't be
based on simple rules yeah but now we
know quantum electrodynamics describes
all of chemistry and and become later
back to that I claim intelligence can be
explained or described in one single
equation this very rich phenomenon you
asked also about whether you know I
understand this phenomenon and it's
probably not and this is saying you
never understand really things you just
get used to them and pretty using used
to sell all automata so you believe that
you understand now why this phenomenon
happens but I give you a different
example I didn't play too much with this
converse game of life but a little bit
more with fractals and with the
Mandelbrot set and it's beautiful you
know patterns just just look Mandelbrot
set and well when the computers were
really slow in our just a black and
white monitor and programmed my own
program sana in assembler - Wow Wow to
get these vectors on the screen and it
was mesmerised and much later so I
returned to this you know every couple
of years and then I try to understand
what is going on and you can understand
a little bit so I try to derive the
locations you know there are these
circles and the Apple shape and then you
have smaller Mandelbrot sets recursively
in this set in this way to
mathematically by solving high order
polynomials to figure out where these
centers are and what size there are
approximately and by sort of ant
mathematically approaching this problem
you slowly get a feeling of why things
are like they are and that sort of isn't
you know first step to understanding why
this rich phenomena do you think as P as
possible what's your intuition you think
it's possible to reverse engineer and
find the short program that generated
the these fractals sort of by what
looking
at the fractals well in principle yes
yeah so I mean in principle what you can
do is you take you know any data set you
know you take these fractals or you take
whatever your data set whatever you have
say a picture of conveys game of life
and you run through all programs you
take your programs 1 2 3 4 and all these
programs around them all in parallel in
so called dovetailing fashion give them
computational resources first one 50%
second 1/2 resources and so on and let
them run wait until they halt give an
output compare it to your data and if
some of these programs produced the
correct data then you stop and then you
have already used some program it may be
a long program because it's faster and
then you continue and you get shorter
and shorter programs until you
eventually find the shortest program the
interesting thing you can never know
whether to short this program because
there could be an even shorter program
which is just even slower and you just
have to wait here but asymptotically and
actually after finite time you have this
shortest program so this is a
theoretical but completely impractical
way of finding the underlying structure
in every data set and there was a lot of
interaction dolls and Kolmogorov
complexity in practice of course we have
to approach the problem more
intelligently and then if you take
resource limitations into account
there's friends the field of
pseudo-random numbers yeah and these are
random that must so these are
deterministic sequences but no algorithm
which is fast fast means runs in
polynomial time can detect that it's
actually deterministic so we can produce
interesting I mean random numbers maybe
not that interesting but just an example
we can produce complex looking data and
we can then prove that no fast algorithm
can detect the underlying pattern which
is unfortunately is it that's a big
challenge for our search for simple
programs in the space of artificial
intelligence perhaps yes it definitely
is quantitative intelligence and it's
quite surprising that it's I can't say
easy here I mean
worked really hard to find his theories
but apparently it was possible for human
minds to find these simple rules in the
universe it could have been different
right it could have been different it's
it's uh it's inspiring so let me ask
another absurdly big question what is
intelligence in your view so I have of
course a definition I wasn't sure what
you're gonna say because you could have
just as easily said I have no clue which
many people would say I'm not modest in
this question so the the informal
version which ever got together be shame
like who co-founded in mind is that
intelligence measures an agent's ability
to perform well in a wide range of
environments so that doesn't sound very
impressive and but it these words have
been very carefully chosen and there is
a mathematical theory behind it and we
come back to that later and if you look
at this this definition right itself it
seems like yeah okay but it seems a lot
of things are missing but if you think
it through then you realize that most
and I claim all of the other traits at
least of rational intelligence which we
usually associate intelligence are
emergent phenomena from this definition
in creativity memorization planning
knowledge you all need that in order to
perform well in a wide range of
environments so you don't have to
explicitly mention that in a definition
interesting so yeah so the consciousness
abstract reasoning or all these kinds of
things are just emerging phenomena that
help you in towards can you say the
definition against multiple environments
did you mention or goals no but we have
an alternative definition instead of
performing value conscious replace it by
goals so intelligence measures an agent
ability to achieve goals in a wide range
of environments that's more or less
because in there there's an injection of
the word goals so you
to specify their there should be a goal
yeah but perform well is sort of what is
it does it mean it's the same problem
yeah there's a little gray area but it's
much closer to something that could be
formalized re in your view are humans
where do humans fit into that definition
are they general intelligence systems
that are able to perform in like how
good are they at fulfilling that
definition at performing well in
multiple environments yeah that's a big
question I mean the humans are
performing best among all species as we
know we know of yeah depends you could
say that trees and plants are doing
better job they'll probably outlast us
so yeah but they're in a much more
narrow environment right I mean you just
you know I have a little bit of air
pollutions and these trees die and we
can adapt right we build houses with
filters we we we do geoengineering so
multiple environment part yes that is
very important yes
so that distinguish narrow intelligence
from wide intelligence also in the AI
research so let me ask the the Alan
Turing question can machines think can
machines be intelligent so in your view
I have to kind of ask the answer is
probably yes but I want to kind of here
with your thoughts on it can machines be
made to fulfill this definition of
intelligence to achieve intelligence
well we are sort of getting there and
you know on a small scale we are already
there the wide range of environments is
missing about yourself driving cars we
have programs which play go and chess we
have speech recognition so it's pretty
amazing but you can you know these are
narrow environments but if you look at
alpha zero that was also developed by
deep mind I mean what famous alphago and
then came alpha zero a year later there
was truly amazing
so on reform a learning algorithm which
is able just by self play to play chess
and then also go and I mean yes they're
both games but they're quite different
games and you know this you didn't don't
feed them the rules of the game and the
most remarkable thing which is still a
mystery to me that usually for any
decent chess program I don't know much
about go you need opening books and
endgame tables and so on - and nothing
in there nothing was put in there it was
alpha zero there's the self play
mechanism starting from scratch being
able to learn actually new strategies is
uh yeah it did rediscovered you know all
these famous openings within four hours
by himself
what I was really happy about I'm a
terrible chess player but I like queen
Gumby and alpha zero figured out that
this is the best opening correct so yes
that you do to answer your question yes
I believe that general intelligence is
possible and it also depends how you
define it do you say AGI with general
intelligence artificial general
intelligence only refers to if you
achieve human-level or a subhuman level
but quite broad is it also general
intelligence so we have to distinguish
or it's only super human intelligence
general artificial intelligence is there
a test in your mind like the Turing test
for natural language or some other test
that would impress the heck out of you
that would kind of cross the line of
your sense of intelligence within the
framework that you said well the Turing
test well has been criticized a lot but
I think it's not as bad as some people
thinking some people think it's too
strong
so it tests not just for a system to be
intelligent but it also has to fake
human deception this section right which
is you know much harder and on the other
hand they say it's too weak yeah because
it just may be fakes
you know emotions or intelligent
behavior it's not real but I don't think
that's the problem
or big problem so if if you would pass
the Turing test
so conversation over terminal with a bot
for an hour or maybe a day or so and you
can fool a human into you know not
knowing whether this is a human or not
that it's
during tests I would be truly impressed
and we have this annual competitions
alumna price and I mean it started with
Elijah that was the first conversational
program and what is it called the
Japanese Mitsouko or so that's the
winner of the last you know a couple of
years and well impressive yes quite
impressive and then google has developed
Meena right just just recently that's an
open domain conversational but just a
couple of weeks ago I think yeah I kind
of like the metric that sort of the
Alexa price has proposed and he maybe
it's obvious to you it wasn't to me of
setting sort of a length of a
conversation like you want the bot to be
sufficiently interestingly you'd want to
keep talking to it for like 20 minutes
and that's a that's a surprisingly
effective in aggregate metric because it
really like nobody has the patience to
be able to talk to about that's not
interesting in intelligent and witty and
is able to go on the different tangents
jump domains be able to you know say
something interesting to maintain your
attention maybe many humans whoops also
fail this test
unfortunately we set just like with
autonomous vehicles with chat BOTS we
also set a bar that's way too hard high
to reach I said you know the Turing test
is not as bad as some people believe you
got what is really not useful about the
Turing test it gives us no guidance how
to develop these systems in the first
place of course you know we can develop
them by trial and error and you know do
whatever and and then run the test and
see whether it works or not but a
mathematical definition of intelligence
gives us you know an objective which we
can then analyze by you know theoretical
tools or computational and you know
maybe improve how close we are and we
will come back to that later with a sexy
model so or I mention the compression
right so in natural language processing
and they have chiefed amazing results
and are one way to test this of course
you know take the system you train it
then you you know see how well it
performs on the task but a lot of
performance measurement is done by so
called perplexity this is essentially
the same as complexity or compression
length so the NLP community develops new
systems and then they measure the
compression length and then they have
ranking and leaks because there's a
strong correlation between compressing
well and then this systems performing
well at the task at hand it's not
perfect but it's good enough for them as
as an intermediate aim
so you mean a measure so this is kind of
almost returning to the coma girl of
complexity so you're saying good
compression usually means good
intelligence yes
so you mentioned you're one of the one
of the only people who dared boldly to
try to formalize our the idea of
artificial general intelligence to have
a a mathematical framework for
intelligence just like as we mentioned
termed IHC AI X I so let me ask the
basic question what is IHC okay so let
me first say what it stands for because
letter stands for actually that's
probably the more basic question but it
the first question is usually how how
it's pronounced but finally I put it on
the website how it's pronounced and you
figured it out yeah
the name comes from AI artificial
intelligence and the X I is the Greek
letter X I which are used for solo
manav's distribution for quite stupid
reasons which I'm not willing to repeat
here in front of camera so it just
happened to be more less arbitrary I
chose to excite but it also has nice
other interpretations so their actions
and perceptions in this model write an
agent his actions and perceptions and
overtime so this is a Index IX index I
so this action at time I and then
followed by reception at time I will go
with that I let it out the first part
yes I'm just kidding I have some
interpretations so at some point maybe
five years ago or ten years ago I
discovered in in Barcelona it wasn't a
big church there wasn't you know stone
engraved some text and the word I see
appeared there I was very surprised and
and and and happy about it and I looked
it up so it is Catalan language and it
means with some interpretation of debts
it that's the right thing to do yeah
Eureka Oh
so it's almost like destined somehow
came yeah yeah came to you in a dream
so Osceola there's a Chinese word I she
also written a galaxy if you could
transcribe that opinion then the final
one is that is AI crossed with induction
because status and that's going more to
the content now so good old-fashioned AI
is more about you know planning and
known data mystic world and induction is
more about often yellow area D data and
inferring models and essentially what
this accident does is combining these
two and I actually also recently I think
heard that in Japanese AI means love so
so if you can combine excise somehow
with that I think we can there might be
some interesting ideas there so I let's
then take the next step can you maybe
talk at the big level of what is this
mathematical framework yeah so it
consists essentially of two parts one is
the learning and induction and
prediction part and the other one is the
planning part
so let's come first to the learning
induction prediction part which
essentially I explained already before
so what we need for any agent to act
well is that it can somehow predict what
happens I mean if you have no idea what
your actions do how can you decide which
acts not good or not so you need to have
some model of what your actions affect
so what you do is you have some
experience you build models like
scientists you know of your experience
then you hope these models are roughly
correct and then you use these models
for prediction and the model is sorry to
interrupt our model is based on you
perception of the world how your actions
will affect that world that's not so
what is the important part but it is
technically important but at this stage
we can just think about predicting say
stock market data whether data or IQ
sequences one two three four five what
comes next yeah so of course our actions
affect what we're doing but I come back
to that in a second so and I'll keep
just interrupting so just to draw a line
between prediction and planning or what
do you mean by prediction in this and
this where it's trying to predict the
environment without your long-term
action in the environment
what is prediction okay if you want to
put the actions in now okay then let's
put in a now yes so the question okay so
this is the simplest form of prediction
is that you just have data which you
passively observe yes and you want to
predict what happens without you know
interfering
as I said weather forecasting stock
market IQ sequences or just anything
okay and Salama of zeref interaction
based on compression so you look for the
shortest program which describes your
data sequence and then you take this
program run it which reproduces your
data sequence by definition and then you
let it continue running and then it will
produce some predictions and you can
rigorously prove that for any prediction
task this is essentially the best
possible predictor of course if there's
a prediction task or tasks which is
unpredictable like you know your fair
coin flips yeah I cannot predict the
next fair country but Solomon of Tarsus
says okay next head is probably 50% it's
the best you can do
so if something is unpredictable Salama
will also not magically predicted but if
there is some pattern and predictability
then Solomonov induction we'll figure
that out eventually and not just
eventually but rather quickly and you
can have proof convergence rates
whatever your data is so there's pure
magic in a sense what's the catch well
the catch is that is not computable and
we come back to that later you cannot
just implement it in even this
Google resources here and run it and you
know predict the stock market and become
rich I mean if ray solomonoff already
not write it at the time but the basic
task is you know you're in the
environment and you're interacting with
an environment to try to learn a model
the environment and the model is in the
space as these all these programs and
your goal is to get a bunch of programs
that are simple and so let's let's go to
the actions now but actually good that
you asked usually I skip this part also
there is also a minor contribution which
I did so the action part but they
usually sort of just jump to the
decision path so let me explain to the
action part now thanks for asking
so you have to modify it a little bit by
now not just predicting a sequence which
just comes to you
but you have an observation then you act
somehow and then you want to predict the
next observation based on the past
observation and your action then you
take the next action you don't care
about predicting it because you're doing
it and then you get the next observation
and you want more before you get it you
want to predict it again based on your
past action and observation sequence
it's just condition extra on your
actions there's an interesting
alternative that you also try to predict
your own actions if you want oh in the
past or the future your future actions
wait let me wrap I think my brain is
broke we should maybe discussed it later
Biff after I've explained the Ising
model it's an interesting variation but
this is a really interesting variation
and a quick comment I don't know if you
want to insert that in here but you're
looking at in terms of observations
you're looking at the entire the big
history a long history of the
observations exactly it's very important
the whole history from birth sort of of
the agent and we can come back to that
I'm also why this is important here
often you know in RL you have MVPs
Markov decision processes which are much
more limiting okay so now we can predict
conditioned on actions so even if the
influenced environment but prediction is
not all we want to do right we also want
to act really in the world and the
question is how to choose the actions
and we don't want to greedily choose the
actions you know
just you know what is best in in the
next time step and we first I should say
you know what is you know how to be
measure performance so we measure
performance by giving the agent reward
that's the so called reinforcement
learning framework so every time step
you can give it a positive reward or
negative reward or baby no reward it
could be a very scarce right like if you
play chess just at the end of the game
you give +1 for winning or -1 for losing
so in the aixi framework that's
completely sufficient so occasionally
you give a reward signal and you ask the
agent to maximise reverb but not
greedily sort of you know the next one
next one because that's very bad in the
long run if you're greedy so but over
the lifetime of the agent so let's
assume the agent lives for M times
that'll say it dies in sort of hundred
years sharp that's just you know the
simplest model to explain so it looks at
the future reward sum and ask what is my
action sequence or actually more
precisely my policy which leads in
expectation because I don't know the
world to the maximum reward some let me
give you an analogy in chess for
instance we know how to play optimally
in theory it's just a minimax strategy I
play the move which seems best to me
under the assumption that the opponent
plays the move which is best for him so
best serve worst for me and the
assumption that he I play again the best
move and then you have this expecting
max three to the end of the game and
then you back propagate and then you get
the best possible move so that is the
optimal strategy which for norman
already figured out a long time ago for
playing adversarial games luckily or
maybe unluckily for the theory it
becomes harder the world is not always
adversarial so it can be if the other
humans even cooperative fear or nature
is usually I mean the dead nature is
stochastic you know you know things just
happen randomly or I don't care about
you so what you have to take into
account is a noise now and not
necessarily Realty so you'll replace the
minimum on the opponent's side by an
expectation which is general enough to
include also the serial cases so now
instead of a minimax trials you have an
expecting max strategy so far so good so
that is well known it's called
sequential decision theory
but the question is on which probability
distribution do you base that if I have
the true probability distribution like
say I play backgammon right there's dice
and there's certain randomness involved
you know I can calculate probabilities
and feed it in the expecting max or the
signature disease we come up is the
optimal decision if I have enough
compute but in the for the real world we
don't know that you know what is the
probability you drive in front of me
brakes and I don't know you know so
depends on all kinds of things and
especially new situations I don't know
so this is this unknown thing about
prediction and there's where solomonoff
comes in so what you do is in sequential
decision jury it just replace the true
distribution which we don't know by this
Universal distribution I didn't
explicitly talk about it but this is
used for universal prediction and plug
it into the sequential decision tree
mechanism and then you get the best of
both worlds you have a long-term
planning agent but it doesn't need to
know anything about the world because
there's a lot of induction part learns
can you explicitly try to describe the
universal distribution and how some of
induction plays a role here yeah I'm
trying to understand so what it does it
I'm so in the simplest case I said take
the shortest program describing your
data run it have a prediction which
would be deterministic yes okay but you
should not just take a shortest program
but also consider the longer ones but
keep it lower a priori probability so in
the Bayesian framework you say a priori
any distribution which is a model or
stochastic program has a certain a
priori probability which is 2 to the
minus and Y to the minus length you know
I could explain length of this program
so longer programs are punished yes a
priori and then you multiplied with the
so-called likelihood function yeah which
is as the name suggests is how likely is
this model given the data at hand so if
you have a very wrong model it's very
unlikely that this model is true so it
is very small number so even if the
model is simple it gets penalized by
that and what you do is then you take
just
the some word this is the average over
it and this gives you a probability
distribution so with universal
distribution of phenomena of
distribution so it's weighed by the
simplicity of the program and likelihood
yes it's kind of a nice idea yeah so
okay and then you said there's you're
playing N or M or forgot the letter
steps into the future so how difficult
is that problem what's involved there
okay so here's a customization problem
what do we do yes so you have a planning
problem up to horizon M and that's
exponential time in in the horizon M
which is I mean it's computable but in
fact intractable I mean even for chess
it's already intractable to do that
exactly and you know it could be also
discounted kind of framework or yes so
so having a heart arising you know at
numbered years it's just for simplicity
of discussing the model and also
sometimes the math is simple but there
are lots of variations actually quite
interesting parameter is its there's
nothing really problematic about it but
it's very interesting so for instance
you think no let's let's then let's let
the parameter M tend to infinity right
you want an agent which lives forever
all right if you do it novel you have
two problems first the mathematics
breaks down because you have an infinite
reward some which may give infinity and
getting river 0.1 in the time step is
infinity and giving you got one every
time service Definity so equally good
not really what we want other problem is
that if you have an infinite life you
can be lazy for as long as you want for
ten years yeah and then catch up with
the same expected reward and you know
think about yourself or you know or
maybe you know some friends or so if
they knew they lived forever you know
why work hard now you know just enjoy
your life you know and then catch up
later so that's another problem with
infinite horizon and you mentioned yes
we can go to discounting but then the
standard discounting is so called
geometric discounting so $1 today is
about worth as much as you know one
dollar and five cents tomorrow so if you
do this so called geometric discounting
you have introduced an effective horizon
so the Aged is now motivated to
had a certain amount of time effectively
it's likely moving horizon and for any
fixed effective horizon there is a
problem to solve which requires larger
horizon so if I look ahead you know five
time steps I'm a terrible chess player
right and I'll need to look ahead longer
if I play go I probably have to look
ahead even longer so for every problem
there forever horizon there is a problem
which this horizon cannot solve yes but
I introduced the so-called near harmonic
horizon which goes down with one or tea
rather than exponential in T which
produces an agent which effectively
looks into the future proportional to
its age so if it's five years old it
plans for five years if it's hundred
years older than plans for hundred years
interesting and a little bit similar to
humans - right and my children don't
plan ahead very long but then we get the
doll - a player I had more longer maybe
when we get all very old I mean we know
that we don't live forever and you're
maybe then how horizon shrinks again so
just adjusting the horizon what is there
some mathematical benefit of that of or
is just a nice I mean intuitively
empirically probably a good idea to sort
of push the horizon back to uh extend
the horizon as you experience more of
the world but is there some mathematical
conclusions here that are beneficial mr.
Loman who talks just a prediction
probably have extremely strong finite
time but no finite data result so you
have sown so much data then you lose on
so much so so the dt r is really great
with the aixi model with the planning
part many results are only asymptotic
which well this is what is asymptotic
means you can prove for instance that in
the long run if the agent you know x
long enough then you know it performs
optimal or some nice things happens so
but you don't know how fast it converges
yeah so it may converge fast but we're
just not able to prove it because a
difficult so that is really dead slow
yeah so so that is what asymptotic means
sort of eventually but we don't know how
fast and if I give the agent a fixed
horizon M
yeah then I cannot prove asymptotic
results right so I mean sort of people
dies in hundred years then and hundred
uses over cannot say eventually so this
is the advantage of the discounting that
I can prove on some topic results so
just to clarify so so I okay I made I've
built up a model well now in a moment
I've have this way of looking several
steps ahead how do I pick what action I
will take it's like with a playing chess
right you do this minimax
in this case here do expect the max
based on the selamat of distribution you
propagate back and then while inaction
falls out the action which maximizes the
future expected reward on the Solano's
distribution and then you just take this
action and then repeat until you get a
new observation and you feed it in this
excellent observation then you repeat
and the reward so on yeah so you're a
row - yeah and then maybe you can even
predict your own action however the idea
but okay this big framework what is it
this is I mean it's kind of a beautiful
mathematical framework to think about
artificial general intelligence what can
you what does it help you into it about
how to build such systems or maybe from
another perspective what does it help us
to in understanding AGI so when I
started in the field I was always
interested two things one was you know
AGI i'm the name didn't exist 10 24th of
january iowa strong AI and physics he
over everything so i switched back and
forth between computer science and
physics quite often you said the theory
of everything the theory of everything
just alike it was a basically the string
of flavors problems before all all of
humanity yeah I can explain if you
wanted some later time you know why I'm
interesting these two questions Nestle
and a small tangent if if if one to be
it was one to be solved which one would
you if one if you were if an apple found
you
head and there was a brilliant insight
and you could arrive at the solution to
one would it be AGI or the theory of
everything
definitely AGI because once the AGI
problem solve they can ask the AGI to
solve the other problem for me yeah
brilliant a put okay so so as you were
saying about it okay so and the reason
why I didn't settle I mean this thought
about you know once we have solved HDI
it solves all kinds of other not just as
here every problem about all kinds of
use more useful problems to humanity
it's very appealing to many people and
you know I thought also that I was quite
disappointed with the state of the art
of the field of AI there was some theory
you know about logical reasoning but I
was never convinced that this will fly
and then there was this Homer more
holistic approaches with neural networks
and I didn't like these heuristics so
and also I didn't have any good idea
myself so that's the reason why I toggle
back and forth quite some violent even
worked some four and a half years and a
company developing software something
completely unrelated but then I had this
idea about the aixi model and so what it
gives you it gives you a gold standard
so I have proven that this is the most
intelligent agents which anybody could
build built in quotation mark right
because it's just mathematical and you
need infinite compute yeah but this is
the limit and this is completely
specified it's not just a framework and
it you know every year tens of
frameworks are developed with just have
skeletons and then pieces are missing
and usually these missing pieces you
know turn out to be really really
difficult and so this is completely and
uniquely defined and we can analyze that
mathematically and we've also developed
some approximations I can talk about it
a little bit later that would dissolve
the top-down approach like say for
Norman's minimax theory that's the
theoretical optimal play of games and
now we need to approximate it put
heuristics in prune the tree blah blah
blah and so on so we can do that also
with an icy body but for generally I
it can also inspire those and most of
most researchers go bottom-up right they
have the systems that try to make it
more general more intelligent it can
inspire in which direction to go what do
you mean by that so if you have some
choice to make right so how should they
evaluate my system if I can't do cross
validation how should I do my learning
if my standard regularization doesn't
work well you know so the answer is
always this we have a system which does
everything that's actually it's just you
know completing the ivory tower
completely useless from a practical
point of view but you can look at it and
see oh yeah maybe you know I can take
some aspects and you know instead of
Kolmogorov complexity there just take
some compressors which has been
developed so far and for the planning
well we have used it here which is also
you know being used in go and it at
least it's inspired me a lot to have
this formal definition and if you look
at other fields you know like I always
come back to physics because I'm a
physics background think about the
Phenom of energy that was long time a
mysterious concept and at some point it
was completely formalized and that
really helped a lot and you can point
out a lot of these things which were
first mysterious and wake and then they
have been rigorously formalized speed
and acceleration has been confused tried
until it was formally defined here there
was a time like this and in people you
know often you know know don't have any
background you know still confused it so
and this is a model or the the
intelligence definitions which is sort
of the dual to it we come back to that
later formalizes the notion of
intelligence uniquely and rigorously so
in in the sense it serves as kind of the
light at the end of the tunnel so before
yeah so I mean there's a million
question I could ask her so maybe the
kind of ok let's feel around in the dark
a little bit so there's been here a deep
mind but in general been a lot of
breakthrough ideas just like we've been
saying around reinforcement learning so
how do you see the progress in
reinforcement learning is different like
which subset of IHC does it occupy
the current like you said the maybe the
Markov assumptions made quite often in
reinforce for learning the there's other
assumptions made in order to make the
system work what do you see is the
difference connection between
reinforcement learning in Nyack see and
so the major difference is that
essentially all other approaches they
make stronger assumptions so in
reinforcement learning the Markov
assumption is that the the next state or
next observation only depends on the on
the previous observation and not the
whole history which makes of course the
mathematics much easier and rather than
dealing with histories of course their
profit from it also because then you
have algorithms that run on current
computers and do something practically
useful but for generally are all the
assumptions which are made by other
approaches we know already now they are
limiting so for instance usually you
need a go digital assumption in the MDP
frameworks in order to learn it goes
this T essentially means that you can
recover from your mistakes and that they
are not traps in the environment and if
you make this assumption then
essentially it can you know go back to a
previous state go there a couple of
times and then learn what what
statistics and what the state is like
and then in the long run perform well in
this state yeah but there are no
fundamental problems but in real life we
know you know there can be one single
action you know one second of being
inattentive while driving a car fast you
know you can ruin the rest of my life I
can become quadriplegic or whatever so
and there's no recovery anymore so the
real world is not err gorica I always
say you know there are traps and there
are situations we are not recover from
and very little theory has been
developed for this case what about what
do you see in there in the context of I
see as the role of exploration sort of
you mentioned you know in the in the
real world and get into trouble when we
make the wrong decisions and really pay
for it but exploration it seems to be
fundamentally
important for learning about this world
for gaining new knowledge so is it his
exploration baked in another way to ask
it what are the parameters of this of
IHC it can be controlled yeah I say the
good thing is that there are no
parameters to control and some other
people track knobs to control and you
can do that I mean you can modify axes
so that you have some knobs to play with
if you want to but the exploration is
directly baked in and that comes from
the Bayesian learning and the long-term
planning
so these together already imply
exploration you can nicely and
explicitly prove that for simple
problems like so-called banded problems
where you say to give a real world
example say you have two medical
treatments a and B you don't know the
effectiveness you try a a little bit be
a little bit but you don't want to harm
too many patients so you have to sort of
trade-off exploring yeah and at some
point you want to explore and you can do
the mathematics and figure out the
optimal strategy it took a Bayesian
agency also non-bayesian agents but it
shows that this Bayesian framework by
taking a prior over possible world's
doing the Bayesian mixture then the
Bayes optimal decision with long term
planning that is important automatically
implies exploration also to the proper
extent not to much exploration and not
too little in this very simple settings
in the IHC model and was also able to
prove that it is a self optimizing
theorem or asymptotic optimality
theorems or later only asymptotic not
finite time bounds it seems like the
long term planning is a really important
but the long term part of the planet is
really important yes and also I mean
maybe a quick tangent how important do
you think is removing the Markov
assumption and looking at the full
history sort of intuitively of course
it's important but is it like
fundamentally transformative to the
entirety of the problem what's your
sense of it like because we all
we make that assumption quite often it's
just throwing away the past now I think
it's absolutely crucial the question is
whether there's a way to deal with it in
a more holistic and still sufficiently
well way so I have to come up with an
example and fly but you know you have
say some you know key event in your life
you know a long time ago you know in
some city or something you realize you
know that's a really dangerous street or
whatever right here and you want to
remember that forever right in case you
come back they're kind of a selective
kind of memory so you remember that all
the important events in the past but
somehow selecting the importance is see
that's very hard yeah and I'm not
concerned about you know just storing
the whole history just you can calculate
you know human life says so you're 100
years doesn't matter right how much data
comes in through the vision system and
the auditory system you compress it a
little bit in this case law silly and
store it we are soon in the means of
just storing it yeah but you still need
to the selection for the planning part
and the compression for the
understanding part the raw storage I'm
really not concerned about and I think
we should just store if you develop an
agent preferably just restore all the
interaction history and then you build
of course models on top of it and you
compress it and you are selective but
occasionally you go back to the old data
and reanalyze it based on your new
experience you have you know sometimes
you you're in school you learn all these
things you think it's totally useless
and you know much later you realize not
you know it looks like as you thought
I'm looking at you linear algebra right
so maybe a minute let me ask about
objective functions because that rewards
it seems to be an important part the
rewards are kind of given to the system
for a lot of people the the
specification of the objective function
is a key part of intelligence like the
the agent itself figuring out what is
important what do you think about that
is it possible within IHC framework to
yourself discover the reward based on
which you should operate okay that'll be
a long answer so and it is a very
interesting question and I asked a lot
about this question where do the rivers
come from and that depends yeah so and
there you know I give you now a couple
of answers so if you want to build
agents now let's start simple so let's
assume we want to build an agent based
on the aixi model which performs a
particular task let's start with
something super simple like I mean super
simple like playing chess or go or
something yeah then you just you know
the reward is you know winning the game
is plus one losing theorems minus one
done you apply this agent if you have
enough compute you let itself play and
it will learn the rules of the game will
play perfect chess
after some while problem solve okay so
if you have more complicated problems
then you may believe that you have the
right rewrote but it's not so a nice
cute example is elevator control that is
also in rich Sutton's book which is a
great book by the way so you control the
elevator and you think well maybe the
reward should be coupled to how long
people wait in front of the elevator you
know long wait is bad you program it and
you do it and what happens is the
elevator eagerly picks up all the people
but never drops them off maybe the time
in the elevator also counts so you
minimize the sum yeah yeah in the
elevator does that but never picks up
the people in the tenth row in the top
floor because in expectation it's not
worth it just let them stay so so even
in apparently simple problems you can
make mistakes
you know and that's what in in war
serious context say a GI safety
researchers consider so now let's go
back to general agents so assume you
want to build an agent which is
generally useful to humans yes we have a
household robot here and it should do
all kinds of tasks so in this case the
human should give the reward on the fly
I mean maybe it's pre trained in the
factory and there there's some sort of
internal reward for you know the battery
level or whatever here but so it you
know it does the dishes badly you know
you punish the robot intercept good you
read what the robot and then train it do
a new task you know like a child right
so you need the human in the loop if you
want a system which is useful to the
human and as long as this agent stays up
human level that should work reasonably
well I'm apart from you know these
examples it becomes critical if they
become you know on a human level it's
it's that miss children small children
you have reason to be well under control
they become older the river technique
doesn't work so well anymore
so then finally so this would be agents
which are just you could sorry slaves to
the humans yeah so if you are more
ambitious and just say we want to build
a new species of intelligent beings we
put them on a new planet and we want
them to develop this planet or whatever
so we don't give them any reward so what
could we do and you could try to you
know come up with some reward functions
like you know it should maintain itself
the robot it should maybe multiply build
more robots right and you know maybe for
all kinds of things did you find useful
but that's pretty hard right you know
what what the self maintenance mean you
know what does it mean to build a copy
should be exact copy an approximate copy
and so that's really hard but LaVon or
so also a deep mind developed a
beautiful model so it just took the aixi
model and coupled the rewards to
information gained so he said the reward
is proportional to how much the agent
had learned about the world and you can
rigorously formally uniquely define it
in terms of our case
versions okay so if you put it in you
get a completely autonomous agent and
actually interestingly for this agent we
can prove much stronger result and for
the general agent which is also nice and
if you let this agent loose it will be
in a sense the optimal scientist is this
absolutely curious to learn as much as
possible about the world and of course
it will also have a lot of instrumental
goals right in order to learn it needs
to at least survive right a dead agent
is not good for anything so it needs to
have self-preservation and if it builds
small helpless acquiring more
information it will do that yeah if
exploration space exploration or
whatever is necessary rights to
gathering information and develop it so
it has a lot of instrumental goals
following on this information gain and
this agent is completely autonomous of
us no rebirth necessary anymore yeah of
course you could define the awaited game
the concept of information it gets stuck
in that library that you mentioned
beforehand with the was a very large
number of books the first agent had this
problem and it would get stuck in front
of an old TV screen which has just said
white noise yeah I know but the second
version can deal with at least
stochasticity well yeah what about
curiosity this kind of word curiosity
creativity is that kind of the reward
function being of getting new
information is that similar to idea of
kind of injecting exploration for its
own sake inside the reward function do
you find this at all appealing
interesting I think that's a nice
definition curiosity is reward sorry
curiosity is exploration for its own
sake yeah I would accept that but most
curiosity well in humans and especially
in children yeah it's not just for its
own sake but for actually learning about
the environment and for behaving so I
would I think most curiosity is tied in
the end towards performing better well
okay so if intelligence systems need to
have the show
function let me you're an intelligent
system currently passing the Turing test
quite effectively what what's the reward
function of our human intelligence
existence what's the reward function
that Marcus hunter is operating under
okay to the first question the
biological reward function is to survive
and to spread and very few humans sort
of are able to overcome this biological
reward function but we live in a very
nice world where we have lots of spare
time and can still survive and spread so
we can develop arbitrary other interests
which is quite interesting on top of
that that yeah but this survival and
spreading sort of is I would say the the
goal or the reward function of human
said that the core one
I like how you avoided answering the
second question which a good
intelligence would so my that your own
meaning of life and the reward function
my own meaning of life and Riyad
function is to find an AGI to build it
beautifully put okay let's dissect Ickes
even further so one of the assumptions
is kind of infinity keeps creeping up
everywhere which what are your thoughts
and kind of bounded rationality and so
the nature of our existence and
intelligence systems is that we're
operating all under constraints under
you know limited time limited resources
how does that how do you think about
that with an IQ framework within trying
to create an eg a system that operates
under these constraints yeah that is one
of the criticisms around I could see
that it ignores computation and
completely and some people believe that
intelligence is inherently tied to
what's bounded resources what do you
think on this one point I think it's do
you think the boundary of resources are
fundamental to intelligence I would say
that an intelligence notion which ignore
computational limits is extremely useful
a good intelligence notion which
includes these resources would be even
more useful but we don't have that yet
and so look at other fields outside of
computer science computational aspects
never play a fundamental role you
develop biological models for cells
something in physics these theories I
mean become more and more crazy and hard
and harder to compute well in the end of
course we need to do something with this
model but this more a nuisance than a
feature and I'm sometimes wondering if
artificial intelligence would not sit in
a computer science department but in a
philosophy department
then this computational focus would be
probably significantly less I mean think
about the induction problem is more in
the philosophy department there's really
no paper who cares about you know how
long it takes to compute the answer
there is completely secondary of course
once we have figured out the first
problem so intelligence without
computational resources then the next
and very good question is could we
improve it by including computational
resources but nobody was able to do that
so far you know even halfway
satisfactory manner I like that that's
in the long run the right department to
belong to this philosophy that's uh it's
really quite a deep idea of or even to
at least to think about big-picture
philosophical questions big-picture
questions even in the computer science
department but you've mentioned
approximation sort of there's a lot of
infinity a lot of huge resources needed
are there approximations - I see that
within the EXCI framework that are
useful you haven't haven't develop a
couple of approximations and what we do
there is that the Sonoma of induction
part which was you know find the
shortest program describe your data we
just replace this by standard data
compressors right and the better
compressors get you know the better this
part will become we focus on a
particular compressor called context
tree weighting which is
pretty amazing lots of well known as
beautiful theoretical properties also
works reasonably well in practice so we
use that for the approximation of the
induction in the learning in the
prediction part and from the planning
part we essentially just took the ideas
from a computer girl from 2006 I was
Java tsipras Perry also now I did mind
who developed the so-called you sit here
algorithm upper confidence bound for
trees algorithm on top of the Monte
Carlo tree search so they approximate is
planning part by sampling and it's
successful on some small toy problems we
don't want to lose the generality all
right and that's sort of the handicap
right if you want to be general you have
to give up something so but this similar
agent was able to play you know small
games like cool poker and tic-tac-toe
and
and even pac-man into the same
architecture no change the agent doesn't
know the rules of the game really
nothing in all by self or by a player
with these environments so your grenade
hoop would propose something called gate
on machines which is a self-improving
program that rewrites its own code well
sort of mathematically philosophically
what's the relationship in your eyes if
you're familiar with it between IHC and
the girl machines yeah familiar with it
he developed it while I was in his lab
you know so the girl machine explained
briefly
you give it a task it could be a simple
task as you know finding prime factors
in numbers right you can formally write
it down there's a very slow algorithm to
do that just all try all the factors
yeah or play chess right optimally you
write the algorithm to minimax to the
end of the game so you write down what
the girdle machine should do then it
will take part of it resources to run
this program and other part of the
sources to improve this program and when
it finds an improved version which
provably
it's the same answer so that's the key
part yeah it needs to prove by itself
that this change of program still
satisfies the original specification and
if it does so then it replaces the
original program by the improved program
and by definition does the same job but
just faster okay and then you know it
proved over it and over it and it's it's
it's developed in a way that all parts
of this girdle machine can self improve
but it stays provably consistent with
the original specification so from this
perspective it has nothing to do with
aixi but if you would now put axial as
the starting axioms in it would run arc
C but you know that takes forever but
then if it finds a provable speed-up of
Arc C it would replace it by this and
that this and this and maybe eventually
it comes up with a model which is still
like C model it cannot be I mean just
for the knowledgeable reader accessing
computable and there can prove that
therefore there cannot be a computable
exact algorithm computers there needs to
be some approximations and this is not
dealt with a good machine so you have to
do something about it but that's the ICT
L model which is finitely computable
which we could put in which part of X is
an non computable the Solomonov
induction part the interaction okay so
but there's ways of getting computable
approximation of the aixi model so then
it's at least computable it is still way
beyond any resources anybody will ever
have but then the girdled machine could
sort of improve it further and further
in an exact way so what this is
theoretically possible that the the girl
machine process could improve isn't
isn't or isn't actually already optimal
it is optimal in terms of the river
collected over its interaction cycles
but it takes infinite time to produce
one action and the world you know
continues whether you want it or not
yeah so the model is assuming had an
Oracle which you know solve this problem
and then in the next hundred
milliseconds or
reaction time you need gives the answer
then ax is optimal
so it's optimal in sense of date are
also from learning efficiency and data
efficiency but not in terms of
computation time and then the other girl
machine in theory but probably not
provably could make it go faster yes ok
interesting those two components are
super interesting the sort of the the
perfect intelligence combined with
self-improvement sort of provable self
improvement since he always liked it
you're always getting the correct answer
and you're improving the beautiful ideas
okay so you've also mentioned that
different kinds of things in in chase of
solving this reward sort of optimizing
for the goal interesting human things
can emerge so is there a place for
consciousness within IHC what where does
uh maybe you can comment because I
suppose we humans are just another
instantiation Vioxx agents and we seem
to have consciousness you say humans are
an instantiation of Mike's agent yes oh
that would be amazing but I think that's
three for the smartest and most rational
humans I think maybe we are very crude
approximation interesting I mean I tend
to believe again I'm Russian so I tend
to believe our flaws are part of the
optimal so the we tend to laugh off and
criticize our flaws and I tend to think
that that's actually close to an optimal
behavior but some flaws if you think
more carefully about it are actually not
floss yeah but I think there are still
enough flaws I don't know it's unclear
as a student of history I think all the
suffering that we've been endured as a
civilization it's possible that that's
the optimal amount of suffering we need
to endure to minimize the long-term
suffering that's your Russian background
that's the Russian weather whoo humans
are or not instantiation of an AI agent
do you think there's a consciousness of
something that could emerge in the
no formal framework like IHC let me also
ask you a question do you think I'm
conscious that's a good question you
you're that that tie is confusing me but
I think you think it makes me
unconscious because it strangles me if
if an agent were to solve the imitation
game posed by touring I think they would
be dressed similarly to you that because
there's a there's a kind of flamboyant
interesting complex behavior pattern
that sells that you're human and you're
cautious but why do you ask was it a yes
always gonna know yes I think you're
conscious yes yeah so and you explain
sort of somehow why
but you infer that from my behavior
right yeah you can never be sure about
that and I think the same thing will
happen with any intelligent way to be
developed if it behaves in a way
sufficiently close to humans or maybe if
not humans I mean you know maybe a dog
is also sometimes a little bit
self-conscious right so so if it behaves
in a way where we attribute typically
consciousness we would actually build
consciousness to this intelligent
systems and you know except all in
particular that of course doesn't answer
the question whether it's really
conscious and that's the you know the
big hard problem of consciousness you
know maybe I'm a zombie I mean not the
movie zombie but the philosophical
zombie it's to you the display of
consciousness close enough to
consciousness from a perspective of a GI
that the distinction of the hard problem
of consciousness is not an interesting
one I think we don't have to worry about
the consciousness problem especially the
heart problem for developing a GI I
think you know we progress at some point
we have solved all the technical
problems and this system will behave
intelligent and then super intelligent
and this consciousness will emerge I
mean definitely it will display behavior
which we will interpret as conscious and
then it's a philosophical question did
this consciousness really emerge or is
zombie which just you know fakes
everything we still don't have to figure
that out although it may be interesting
at least from a philosophical point of
it's very interesting but it may also be
sort of practically interesting you know
there's some people say you know if it's
just faking consciousness and feelings
you know then we don't need to have be
concerned about you know rights but if
it's real conscious and has feelings
then we need to be concerned yeah I
can't wait til the day where AI systems
exhibit consciousness because it'll
truly be some of the hardest ethical
questions how well we do with that it is
rather easy to build systems which
people ascribe consciousness and I give
you an analogy I mean remember maybe
once before you were born the Tamagotchi
yes how dare you sir you're young right
yes it's good thing yeah thank you thank
you very much but I was also in the so
you have any of those funny things but
you have heard about this time ago it
was you know really really primitive
actually for the time it was and you
know you could race you know this and
and and and kids got so attached to it
and you know didn't want to let it die
and would have probably if we would have
asked you know the children know do you
think this drama coach is conscious and
they would say yes yes I was yes that's
kind of a beautiful thing actually
because that consciousness ascribing
consciousness seems to create a deeper
connection yeah which is a powerful
thing but we have to be careful on the
ethics side of that well let me ask
about the AGI community broadly you kind
of represent some of the most serious
work on a giass of at least or earlier
and deepmind represents a serious work
on AGI these days but why in your sense
is the AGI communities so small or has
been so small until maybe deep mine came
along like why why aren't more people
seriously working on human level and
super human level intelligence from a
formal perspective okay from a formal
perspective that sort of you know and an
extra point so I think
a couple of reasons I mean AI came in
waves right you know our interest in our
summers and then there were big promises
which were not fulfilled and people got
disappointed and that narrow AI are sold
in particular problems which seem to
require intelligence was always to some
extent successful and there were
improvements small steps and if you
build something which is you know useful
for society or industrial useful then
there's a lot of funding so I guess it
wasn't pass the money which drives
people to develop specific system
solving specific tasks but you would
think that you know at least on
university you should be able to do
ivory tower research and that was
probably better a long time ago about
even nowadays there's quite some
pressure off of doing applied research
or translational research and you know
it's harder to get grants as a theorist
so that also drives people away it's
maybe also harder attacking the general
intelligence problem so I think enough
people I mean maybe a small number we're
still interested in in formalizing
intelligence and thinking of general
intelligence but you know not much came
up right or not much great stuff came up
so what do you think we talked about the
formal big light at the end of the
tunnel but from the engineering
perspective what do you think it takes
to build an a GI system is it and I
don't know if that's a stupid question
or a distinct question from everything
we've been talking about I exceed but
what do you see as the steps that are
necessary to take to start to try to
build something so you wanted a blue
print now and then you go and do it it's
the whole point of this conversation try
to squeeze that in there now is there I
mean what's your intuition is it is in
the robotic space or something that has
a body and tries to explore the world is
in the reinforcement learning space like
the efforts of the alpha 0 and alpha
star they're kind of exploring how you
can solve it through in in the
simulation in the gaming world
their stuff and sort of the of the
transformer working natural English
processing so maybe attacking the open
domain dialog like what where do you see
a promising pathways let me pick the
embodiment maybe so embodiment is
important yes and no I don't believe
that we need a physical robots walking
or rolling around interacting with the
real world in order to achieve AGI and I
think it's more of a distraction
probably than helpful it's sort of
confusing the body with the mind for
industrial applications or near-term
applications of course we need robotics
for all kinds of things yeah but for
solving the big problem at least at this
stage I think it's not necessary but the
answer is also yes that I think the most
promising approaches that you have an
agent and you know there can be a
virtual agent you know you know computer
interacting with an environment possibly
in our 3d simulated environment like in
many computer games and and you train
and learn the agent even if you don't
intend to later put it sort of you know
this algorithm in a robot brain and
leave it forever in the virtual reality
getting experience in a also it's just
simulated 3d world is possibly and I say
possibly important to understand things
on a similar level as humans do
especially if the agent or primarily if
the agent wants needs to interact with
the humans right you know if you talk
about objects on top of each other in
space and flying and cars and so on and
the agent has no experience with even
virtual 3d worlds it's probably hard to
grasp so if you develop an abstract
agent say we take the mathematical path
and we just want to build an agent which
can prove theorems and becomes a better
imitation then this agent needs to be
able to reason in very abstract spaces
and then maybe sort of putting it into
3d environment simulated alt is even
harmful it should sort of you put it in
I don't know an environment which it
creates itself or so it seems like you
have an interesting rich complex
trajectory through life in terms of your
journey of ideas so it's interesting to
ask what books technical fiction
philosophical and books ideas people had
a transformative effect books are most
interesting because maybe people could
also read those books and see if they
could be inspired as well you're luckily
asked books and not singular book it's
very hard and I tried to pin down one
book yeah then I can do that at the end
so the most the books which were most
transformative for me or which I can
most highly recommend to people
interested in AI both perhaps yeah I
would always start with Russell and
Norvig artificial intelligence a modern
approach that's the AI Bible it's an
amazing book it's very broad it covers
you know all approaches to AI and even
if you focus on one approach I think
that is the minimum you should know
about the other approaches out there so
that should be your first book
fourth edition should be coming out soon
okay interesting deeper there's a deep
learning chapter now so there must be
written by Ian good fella okay and then
the next book I would recommend the
reinforcement only book by certain in
part oh there's a beautiful book if
there's any problem with the book
it makes our L feel and look much easier
than it actually is it's very gentle
book it's very nice to read the
exercises do you can very quickly you
know get some aerial systems to run you
know on very toy problems but it's a lot
of fun and you in very in a couple of
days you feel you know you know what RL
is about but it's much harder than the
book yeah
come on now it's an awesome book yeah
that idea's yeah and maybe I mean
there's so many books out there if you
like the information theoretic approach
then there's Kolmogorov complexity by
Alene batani but probably you know some
some short article is enough you don't
need to read a whole book but it's a
great book and if you have to mention
one all-time favorite book so different
flavor that's a book which is used in
the International Baccalaureate for high
school students in several countries
that's from Nicolas alchun theory of
knowledge second edition or first not
assert least the third one they put they
took out all the fun okay so this asked
all the interesting or to me interesting
philosophical questions about how we
acquire knowledge from all perspectives
on from math from art from physics and
ask how can we know I'm anything and
book is called theory of knowledge from
which is almost like a philosophical
exploration of how we get knowledge from
anything yes yeah I mean can religion
tell us you know about something about
the world can science tell us something
about the world can mathematics so as
it's just playing with symbols and
onions
open-ended questions and I mean it's for
high school students so they have been
resources from Hitchhiker's Guide to the
galaxy and from Star Wars and the
chicken cross the road
yeah and it's it's it's fun to read and
but it's also quite deep if you could
live one day of your life over again
because it made you truly happy or maybe
like we said with the books it was truly
transformative what what day what moment
would you choose there's something pop
into your mind doesn't need to be a day
in the past or can it be a day in the
future
well space-time is an emergent phenomena
so it's all the same anyway
okay okay from the past you're really
good saved from the future I love it no
I will also tell you from the future
okay from the past I would say
when I discovered Maxim Allah I mean it
was not in one day but it was one
moment they are realized comig of
complexity and didn't even know that it
existed but I rediscovered sort of this
compression idea myself but immediately
I knew I can't be the first one but I
had this idea and then I knew about
sequential decision ray and I knew if I
put it together this is the right thing
and yeah I'm still when I think back
about this moment I'm I'm super excited
about it was there
was there any more details and context
that moment did an apple fall in your
head were so like if you look at en
Goodfellow talking about Gans there was
beer involved there is there some more
context of what sparked your thought it
was a jest and no it was much more
mundane so I've worked in this company
so in this sense the four and a half
years was not completely wasted so and
I've worked on an image interpolation
problem and I developed a quite neat new
interpolation techniques and they got
patented and then I you know and which
happens quite often I got sort of
overboard and thought about you know
yeah that's pretty good but it's not the
best so what is the best possible way of
doing in the interpolation and then I
thought yeah you you want the simplest
picture which is if you cross train it
recovers your original picture and then
I you know thought about the simplicity
concept more in quantitative terms and
you know then everything developed and
somehow love the full beautiful mix of
also being a physicist and thinking
about the big picture of it then led you
to probably the end of a good idea so as
a physicist I was probably trained not
to always think in computational terms
you know just ignore that and think
about the other two the fundamental
properties which you want to have so
what about if you could really one day
in the future all the day what would
that be when I solve the AGI problem and
I bring the practice in practice so in
theory I have solved it that I see what
already attracted me and then ask the
first question or would be the first
question what's the meaning of life
I don't think there's a better way to
end it thank you so much for talking it
is a huge honor to finally meet you yeah
thank you - I was a pleasure off my side
- thanks for listening to this
conversation with Marcus hunter and
thank you to our presenting sponsor cash
app downloaded you just cold legs
podcast you'll get ten dollars and ten
dollars will go to first an organization
that inspires and educates young minds
to become science and technology
innovators of tomorrow if you enjoy this
podcast subscribe on YouTube give it
five stars an apple podcast supported on
patreon or simply connect with me on
Twitter at Lex Friedman and now let me
leave you with some words of wisdom from
Albert Einstein the measure of
intelligence is the ability to change
for listening and hope to see you next
time
you