Kind: captions
Language: en
a lot of people have said for many years
that there will come a time when they
want to pause a little bit
that time is now
the following is a conversation with Max
tegmark his third time in the podcast in
fact his first appearance was episode
number one of this very podcast he is a
physicist and artificial intelligence
researcher at MIT co-founder of future
Life Institute and author of Life 3.0
Being Human in the age of artificial
intelligence
most recently he's a key figure in
spearheading the open letter calling for
a six-month pause on giant AI
experiments like training gpt4 the
letter reads
were calling for a pause on training of
models larger than GPT 4 for 6 months
this does not imply a pause or ban on
all AI research and development or the
use of systems that have already been
placed on the market our call is
specific and addresses a very small pool
of actors who possesses this capability
the letter has been signed by over 50
000 individuals including 1800 CEOs and
over 1500 professors signatories include
Joshua bengio Stewart Russell Elon Musk
Steve Wozniak you all know a Harari
Andrew Yang and many others
this is a defining moment in the history
of human civilization or the balance of
power between human and AI begins to
shift
and Max's mind and his voice is one of
the most valuable and Powerful in a time
like this
his support his wisdom his friendship
has been a gift of forever deeply
grateful for
this is the Lex Friedman podcast to
support it please check out our sponsors
in the description and now dear friends
here's Max Ted mark
you were the first ever guest on this
podcast episode number one so first of
all
Max I just have to say uh thank you for
giving me a chance thank you for
starting this journey it's been an
incredible journey just thank you for um
sitting down with me and just acting
like I'm somebody who matters that I'm
somebody who's interesting to talk to
and uh thank you for doing it I meant a
lot all right thanks to you for putting
your heart and soul into this I know
when you delve into controversial topics
it's inevitable to get hit by what what
Hamlet talks about the slings and arrows
and stuff and I really admire this it's
in an era you know where YouTube videos
are too long and now it has to be like a
20 minute Tick Tock 20 second Tick Tock
clip it's just so refreshing to see you
going exactly against all of the advice
and doing this with this really long
form things and that people appreciate
it you know reality is nuanced and uh
thanks for
and sharing it that way uh so let me ask
you again the first question I've ever
asked on this podcast episode number one
talking to you do you think there's
intelligent life out there in the
universe let's revisit that question do
you have any updates
what's your view
when you look out to the Stars so when
we look after the Stars
if you define our universe the way most
astrophysicists do not this all of space
but the spherical region of space that
we can see with our telescopes from
which light has the time to reach us
since our big bang
I'm in the minority I I'm
estimate that we are the only life
in this spherical volume that has uh
invented Internet radios gotten our
level of tech and um if that's true
then it puts a lot of responsibility on
us to not mess this one up because if
it's true it means that life is is quite
rare and we are stewards of this one
spark of advanced Consciousness which if
we nurture it then
help it grow it immensely life can
spread from here out into much of our
universe and we can have this just
amazing future whereas if we instead um
are Reckless with the technology we
build and just snuff it out due to the
stupidity
or in fighting then
maybe the rest of cosmic history in our
universe was just going to be a play for
empty benches but I I do think
that we are actually very likely to get
visited by aliens
alien intelligence quite soon but I
think we are going to be building that
alien intelligence
so
uh we're going to give birth
to an intelligent alien civilization
unlike anything that human the evolution
here on Earth was able to create in
terms of the path the biological path it
took yeah and it's gonna be much more
alien than
cats or even the most exotic animal on
the planet right now because it will not
have been created through the usual
darwinian competition where it
necessarily cares about
self-preservation is afraid of death
um any of those things
the space of alien Minds is just that
you can build it's just so much faster
than what evolution will give you
and with that also comes great
responsibility but the the for us to
make sure that the kind of Minds we
create are those kind of Minds that um
it's good to create Minds that will uh
share our values and and be good for
Humanity and life and also mine don't
create Minds that don't suffer
do you try to visualize the full space
of alien Minds that AI could be to try
to consider all the different kinds of
intelligences
sort of generalizing what humans are
able to do to the full spectrum of an
intelligent creatures entities could do
I try but I would say I fail I mean it's
it's very difficult for a human mind
really grapple with
something still completely alien even
even for us right if we just try to
imagine how would it feel if we were
completely indifferent towards death or
individuality if we even if you just
imagine that for example
you could just copy my knowledge of how
to speak Swedish boom now you can speak
Swedish
and you could copy any of my cool
experiences and then you could delete
the ones you didn't like in your own
life just like that it
it would already change quite a lot
about how you feel as a human being
right if you probably spend less effort
studying things if you just copy them
and you might be less afraid of death
because if the plane you're on starts to
crash you'd just be like oh shucks I'm
gonna I haven't backed my brain up for
four hours so I'm gonna lose this all
this wonderful experiences of this
flight
we might also start feeling
more like compassionate maybe with other
people if we can so readily share each
other's experiences in our knowledge and
feel more like a hive mind it's very
hard though I I really
feel very humble about this
to to Grapple with it that the how it
might actually feel that the the one
thing which is so obvious though it's I
think is just really worth reflecting on
is because the mind space of possible
intelligence is so different from ours
it's very dangerous if we assume they're
going to be like us or anything like us
well there's a
the entirety of uh human written history
has been through poetry through novels
been trying to describe her philosophy
uh try and describe the Human Condition
and what's entailed in it like just like
you said fear of death and all those
kinds of things what is love and all of
that changes yeah if you have a
different kind of intelligence yeah like
all of it the entirety all those poems
they're trying to sneak up to what the
hell it means to be human all of that
changes how AI concerns and uh
existential crises that AI experiences
how that clashes with the human
existential crisis The Human Condition
yeah that's hard to hard to Fathom how
to predict it's hard but it's
fascinating to think about also even in
the best case scenario where we don't
lose control of over the ever more
powerful AI that we're building to
other humans whose goals we think are
horrible and where we don't lose control
to the machines
and AI
provides the things we want even then
you get into the questions do you
touched here you know maybe it's the
struggle that it's actually hard to do
things is part of the things that gives
us meaning as well right so for example
I found it so shocking that this new
Microsoft gpt4 commercial that they put
together has this woman talking about
and showing this demo how she's going to
give a graduation speech to her beloved
daughter and she asks gpt4 to write it
it was freaking 200 words or so if I
realized that my parents couldn't be
bothered struggling a little bit to
write
200 words and Outsource that to their
computer I would feel really offended
actually
and so I wonder if um eliminating too
much of the struggle from our existence
what
do you think that would also take away a
little bit of
what it means to be human yeah
we can't even predict I had somebody
mentioned to me that they use they
started using uh
GPT with a 3.5 not 4.0
uh to write what they really feel to a
person
and they have a temper issue and they're
basically trying to get Chad gbt to
rewrite it in a nicer way to get the
point across but we write in a nicer way
so we're even removing the inner
from our communication so I don't you
know there's some positive
aspects of that but mostly it's just the
transformation of how humans communicate
and it's scary because so much of our
society is based on this glue of
communication and if that we're now
using AI as the medium of communication
that that does the language for us uh so
much of the emotion that's Laden in
human communication so much of the
intent
that's going to be handled by an
outsourced AI how does that change
everything how how does it change the
internal state of how we feel about
other human beings what makes us lonely
what makes us excited yeah what makes us
afraid how we fall in love all that kind
of stuff yeah for me personally I have
to confess the challenge is one of the
things that really makes my life feel
meaningful you know
if I go hiking mountain with my wife
Maya I don't want to just press a button
and be at the top but I want to struggle
and come up there sweaty and feel wow we
did this in the same way
I want to constantly work on myself to
become a better person if I say
something in Anger that I regret I want
to go back and and really work on myself
rather than just tell an AI just from
now on always filter what I write so I
don't have to work on myself because
then I'm not growing
yeah but then again it could be like
with chess and AI
wants it significantly obviously
supersedes the performance of humans it
will live in its own world and provide
maybe a uh flourishing Civilizations for
humans but we humans will continue
hiking mountains and playing our games
even though AI is so much smarter so
much stronger so much Superior in every
single way just like with chess yeah so
that that I mean that's one possible
hopeful trajectory here is that humans
will continue to Human
uh and AI will just be a
kind of
a a medium that enables The Human
Experience to flourish yeah
I would phrase that as rebranding
ourselves from Homo sapiens to homo
sentience you know right now with
sapiens the ability to be intelligent
we've even put it in our species name
we're branding ourselves as the smartest
yeah information processing
entity on the planet that's clearly
gonna
change if AI continues ahead
so maybe we should focus on the
experience instead the subjective
experience that we have with
homo sentience and and so that's what's
really valuable the love the connection
the other things
and get off our high horses and get rid
of this hubris that only we can
do we do integrals so Consciousness and
subjective experience is a fundamental
value
to what it means to be human make that
make that the priority
that feels like a hopeful direction to
me but that also requires more
compassion not just towards other humans
because they happen to be the smartest
on the planet but also towards all our
other fellow creatures on this planet
and I I personally feel right now we're
treating a lot of farm animals horribly
for example and the excuse we're using
is oh they're not as smart as us
but if we get that we're not that smart
in the grand scheme of things either in
the post aie Epoch you know
then surely we should value
the subjective experience of a cow also
well
allow me to briefly look at the book
which at this point is becoming more and
more Visionary that you've written I
guess over five years ago life 3.0
so first of all 3.0 what's 1.0 what's
2.0 was 3.0 and how's that Vision sort
of evolve the vision in the book evolved
to today life 1.0 is really dumb like
bacteria and that it can't actually
learn anything at all during the
lifetime the learning just comes from
this genetic
process from one generation to the next
Life 2.0 Is Us and other animals which
have brains which can learn during their
lifetime a great deal right so
and um you know you were born without
being able to speak English and at some
point you decided hey I want to upgrade
my software let's install an
English-speaking module
so you did
and life 3.0 does not exist yet can
cannot replace not only its software the
way we can but also it's Hardware
and um that's where we're heading
towards at high speed we're already
maybe 2.1 because we can you know put in
a
an artificial knee uh pacemaker
etc etc and if newer Link in other
companies succeed will be like 2.2 Etc
but uh well the Company's trying to
build AGI are trying to make is of
course full 3.0 and you can put that
intelligence into something that also
has no
biological basis whatsoever so let's
constraints and more capabilities just
like the leap from 1.0 to 2.0 there is
nevertheless he's speaking so harshly
about bacteria so disrespectfully about
bacteria there is still the same kind of
magic there
that permeates Life 2.0 and uh and 3.0
it seems like maybe the thing that's
truly powerful
about life intelligence and
Consciousness was already there in 1.0
is it possible
I think we should be humble and not be
so quick
make everything binary and say either
it's there or it's not clearly there's a
there's a great spectrum and there is
even controversy by whether some
unicellular or organisms like amoebas
can maybe learn a little bit
you know after all so apologies if I
offended anything there yeah it wasn't
by intent it was more that I wanted to
talk up how cool it is to actually have
a brain yeah where you can learn
dramatically within your lifetime
typical human and and the higher up you
get from 1.0 to 2.0 to 3.0 the more you
become the captain of your own desk of
your own ship the master of your own
destiny and the less you become a slave
to whatever Evolution gave you right
by upgrading our software which can be
so different from previous generations
and even from our parents
much more so than even a bacterium you
know no offense to them
and if you can also swap out your
Hardware take any physical form you want
of course it's really the sky's the
limit
yeah so the
it accelerates the rate at which you can
perform the competition computation that
determines your destiny
yeah and I think it's it's worth
commenting a bit on what you means in
this context also if you swap things out
a lot right now
this is controversial but my
current
understanding is that that you know
life is best thought of not as a bag of
meat or even a bag of
Elementary particles but rather as in as
um
a system which can process information
and retain its own complexity
even though nature is always trying to
mess it up so
it's all about information processing
and
that makes it a lot like something like
a wave in the ocean which is not it's
it's water molecules right the water
molecules bob up and down but the wave
moves forward it's an information
pattern in the same way you Lex
you're not the same atoms as during the
first time you did with me you've
swapped out most of them but still you
yeah and
the the information pattern is still
there and um
if you if you could swap out your arms
and whatever
you can still have this kind of
continuity it becomes much more
sophisticated sort of way before in time
where the information lives on I I lost
both of my parents since since our last
podcast and and it actually gives me a
lot of Solace that
this way of thinking about them
they haven't entirely died because
a lot of mommy and daddy's um
sorry I'm getting a little emotional
here but a lot of their values
and ideas and even jokes and so on they
haven't gone away right some of them
live on I can carry on some of them and
they also live on a lot of other and a
lot of other people so in this sense
even with Life 2.0 we can to some extent
already transcend
our physical bodies and our death
and particularly if you can share your
own information your own ideas with many
others like you do in your podcast
then um
you know that's the closest immortality
we can get with our biobodies you carry
a little bit of them in you yes yeah
uh do you miss them you miss your mom
and dad of course of course what did you
learn about life from them if it can
take a bit
of a tangent
on so many things
um
for starters my my Fascination for Math
and
um the physical mysteries of our
University thinking I got a lot of that
for my dad but I think my obsession for
really big questions and Consciousness
and so on that actually came mostly for
my mom
and
when I got from both of them which is
very core part of really who I am I
think is
is
um
this um
just feeling comfortable with
not buying into what everybody else is
saying just
dude what I think is right
they both
very much just you know did their own
thing and sometimes they got flagged for
it and it did it anyway
that's why you've always been an
inspiration to me that you're at the top
of your field and you still
you still willing to uh
to tackle the big questions in your own
way you're one of the one of the people
that represents
MIT best to me you've always been an
inspiration in that so it's good to hear
that you got that from your mom and dad
yeah you're too kind but but yeah I mean
the real the good reason to do science
is because you're really curious you
want to figure out the truth
if you think
this is how it is and everyone else says
no no that's and it's that way
you know
you sticked with what you think is true
and and
even if
everybody else keeps thinking it's
there's a certain
um
I always root for the underdog when I
watch movies and my my dad once I I one
time for example when I wrote one of my
craziest papers ever or I'm talking
about our universe ultimately being
mathematical which we're not going to
get into today I got this email from a
quite famous Professor saying this is
not only but it's going to ruin
your career you should stop doing this
kind of stuff I sent it to my dad do you
know what he said what'd he say he
replied with a quote from Dante segil Tu
Corso
follow your own path and let the people
talk
go Dad yeah this is the kind of thing
you know he's dead but that that
attitude is not
how did losing them as a man as a human
being change you
how did it expand your thinking about
the world how did it uh expand your
thinking about
you know this thing we're talking about
which is humans creating another living
sentient perhaps uh being
I think it uh
mainly do two things uh
one of them just going through all their
stuff after they had passed away and so
on just drove home to me how important
it is to ask ourselves
why are we doing this things we do
because it's inevitable that you look at
some things they spent an enormous time
on and you asked in hindsight would they
really have spent so much time on this
or if would they have done something
that was more meaningful
um so I've been looking more in my life
now and asking you know why am I doing
what I'm doing and I I feel
it should either be something I really
enjoy doing or it should be something
that I find really really meaningful
because it helps
Humanity
and um
if it's in none of those two categories
maybe I should spend less time on it you
know the other thing is dealing with
death up in personal like this it's
actually made me less afraid
of
um
even less afraid of other people telling
me that I'm an idiot you know which
happens regularly and just live my life
do my thing you know
um
and um
it's made it a little bit easier for me
to focus on what I what I feel is really
important what about fear of your own
death
has it made it more real that this is
that this is something that happens yeah
it's made it extremely real and I'm next
next in line in our family now right
it's me and my brother my younger
brother but um
they both handled it with such dignity
it was there was a true inspiration also
they never complained about things and
you know when you're old and your body
starts falling apart it's more and more
to complain about they looked at what
could they still do that was meaningful
and they focused on that rather than
wasting time
talking about or even thinking much
about things they were disappointed in
I think anyone can make themselves
depressed if they start their morning by
making a list of grievances
whereas if you start your day and when
the little meditation and just the
things you're grateful for you you
basically choose to be a happy person
because you only have a finite number of
days you should spend them Make It Count
being grateful yeah
well you do happen to be working on a
thing which seems to have a potentially
some of the greatest impact on human
civilization of anything humans have
ever created which is artificial
intelligence this is on the both
detailed technical level and in the high
philosophical level you work on so
you've mentioned to me that there's an
open letter
that you're working on it's actually uh
going live in a few hours so I've been
having late nights and early mornings
it's been very exciting actually I in
short
I have you seen uh don't look up
the film
yes yes I don't want to be the movie
spoiler for anyone watching this who
hasn't seen it but if you're watching
this you haven't seen it watch it
because we are actually acting out it's
it's life imitating art humanity is
doing exactly that right now except
it's an asteroid that we are building
ourselves
almost nobody is talking about it
people are squabbling across the planet
about all sorts of things which seem
very minor compared to the asteroid
that's about to hit us right uh most
politicians don't even have their radar
this on the radar they think maybe in
100 years or whatever
right now
we're at a fork on the road this is the
most important um Fork the humanity has
reached in its over a hundred thousand
years on this planet we're building
effectively a new species that's smarter
than us
it doesn't look so much like a species
yet because it's mostly not embodied in
robots but um
that's a technicality which will soon be
changed and and this arrival of
artificial general intelligence that can
do all our jobs as well as us and
probably shortly thereafter super
intelligence which greatly exceeds our
cognitive abilities it's going to either
be the the best thing ever to happen
Humanity or the worst I'm really quite
confident that there is
not that much Middle Ground there but it
would be fundamentally transformative
to human civilization of course utterly
and totally you know again we branded
ourselves as Homo sapiens because it
seemed like the basic thing where the
King of the castle on this planet were
the Smart Ones if we can control
everything else
this could very easily change we're
certainly not going to be the smartest
on the planet for very long if AI unless
AI progress just Falls and we can talk
more about why I I think that's true
because it's it's controversial
and and then we can also talk about
reasons we might think it's gonna be the
best thing ever and the reason you think
it's going to be the end of humanity
which is of course super controversial
but
what I think we can anyone who's working
on uh Advanced AI
can agree on is it's it's much like the
film don't look up and that
it's just really comical how little
serious public debate there is about it
given how huge it is
so what we're talking about is the
development of currently things like
gpt4
and the signs it's showing of uh rapid
Improvement that may in the near term
lead to development of super intelligent
AGI AI General AI systems and what kind
of impact that has on society exactly
when that thing is achieves General
human level intelligence and then beyond
that General superhuman level
intelligence
there's a lot of questions to explore
here so one you mentioned halt is that
uh the content of the letter is to
suggest that maybe we should pause the
development of these systems exactly so
this is very controversial
from
when we talked the first time we talked
about how I was involved in starting the
future Life Institute and we worked very
hard on 2014-2015 was the mainstream AI
safety
the idea that there even could be risks
and that you could do things about them
before then a lot of people thought it
was just really kooky to even talk about
it and a lot of AI researchers felt
worried that this was too flaky and
could be bad for funding and that the
people had talked about it or just not
didn't understand AI
I'm very very happy with
how that's gone in that now you know
just completely mainstream you go on any
AI conference and people talk about AI
safety and it's a nerdy technical field
full of equations and simula and blah
blah yes
um
as it should be uh
but there's this other thing which has
been quite taboo up until now
calling for slowdown so what
we've been constantly been saying
including myself I've been biting my
tongue a lot you know is that you know
we we don't need to slow down AI
development we just need to win this
race the wisdom race between the growing
power of the AI and the growing wisdom
with which we manage it and rather than
trying to slow down AI let's just try to
accelerate the wisdom do all this
technical work to figure out how you can
actually ensure that your powerful AI is
going to do what you wanted to do and
have Society adapt also
with um incentives and regulations so
that these things get put to good use
um sadly that
didn't pan out
the progress on technical Ai and
capabilities has gone a lot faster than
than many people thought
back when we started this in 2014 turned
out to be easier to build really
Advanced AI than we thought
um
and on the other side it's gone much
slower than we hoped with getting
um
policy makers and others to actually
put them incentives in place to to make
steer this in the in the good directions
we can maybe we should unpack it and
talk a little bit about each so yeah why
did it go faster than we than a lot of
people thought them
in hindsight it's exactly like building
um
flying machines
people spent a lot of time wondering
about how the birds fly you know and
that turned out to be really hard have
you seen the Ted talk with a flying bird
like a flying robotic Bird yeah it flies
around the audience but it took a
hundred years longer to figure out how
to do that than for the Wright brothers
to build the first airplane because it
turned out there was a much easier way
to fly
and evolution picked a more complicated
one because it had its hands tied it
could only build a machine that could
assemble itself which the Wright
brothers didn't care about they can only
build a machine they'll use only the
most common atoms in the periodic table
Wright brothers didn't care about that
they could use steel
iron atoms and it had to be able to
repair itself and it also had to be
incredibly fuel efficient you know
a lot of birds use less than half the
fuel of a remote control plane that's
flying the same distance
for humans let's throw a little more put
a little more fuel in a roof there you
go 100 years earlier
that's exactly what's happening now with
these large language models
the brain is incredibly complicated
many people made the mistake you're
thinking we had to figure out how the
brain does human level AI first before
we could build in the machine
that was completely wrong you can take
an incredibly simple
computational system called the
Transformer Network and just train it to
do something incredibly dumb
just read a gigantic amount of text and
try to predict the next word
and it turns out
if you just throw a ton of compute at
that and a ton of data it gets to be
frighteningly good like gpt4 which I've
been playing with so much since it came
out right
and um
there's still some debate about whether
that can get you all the way to full
human level or not
but uh yeah we can come back to the
details of that and how you might get
the human level AI even if
a large language models don't
can you briefly if it's just a small
tangent comment on your feelings about
gpt4 so just that you're impressed by
this rate of progress but where where is
it can gpt4 reason
what are like the intuitions what are
human interpretable words you can assign
to the capabilities of gpt4 that makes
you so damn impressed with it I'm both
very excited about it and terrified
interesting mixture of promotions all
the best things in life include those
two somehow yeah I can absolutely reason
anyone who hasn't played with it I
highly recommend doing that before
dissing it
it can do quite quite remarkable
reasoning and
I've had to do a lot of things which I
realized I couldn't do that myself that
well even and and obviously does it
dramatically faster than we do too when
you watch it type
and it's doing that while servicing a
massive number of other humans at the
same time at the same time it cannot
reason
as well as a human can on some tasks
just because it's obviously a limitation
from its architecture you know we have
in our heads what in geekspeak is called
the recurrent neural network there are
Loops information can go from this
neuron the base neuron to this neuron
and then back to this one you can like
ruminate on something for a while you
can self-reflect a lot uh these large
language models that are they cannot
like gpt4 it's it's a so-called
Transformer where it's just like a
one-way Street of information basically
and geekspeak it's called the feed
forward neural network
and it's only so deep so it can only do
logic that's that many steps and that
deep and it's not
and you can so you can create problems
which will fail to solve you know for
that reason
um
but the fact that it can do so amazing
things with this incredible simple
architecture already it's quite stunning
and and what we see in my lab at MIT
when we look inside
large language models to try to figure
out how they're doing it that's the key
core focus of our research it's called
um mechanistic interpretability in geek
speak you know you have this machine it
does something smart you try to reverse
reverse engineer see how does it do it
are you think of it also as artificial
Neuroscience that's exactly what
neuroscientists do with actual brains
but here you have the advantage that you
can you don't have to worry about
measurement errors you can see what
every neuron is doing all the time and
and a recurrent thing we see again and
again
there's been a number of beautiful
papers quite recently by
by a lot of researchers some of them
here I am in this area is where when
they figure out how something is done
you can say oh man that's such a dumb
way of doing it and you immediately see
how it can be improved like for example
there was a beautiful paper recently
where they figured out how a large
language model stores certain facts like
Eiffel Towers in Paris
and they figured out exactly how it's
stored and where the proof that they
understood it was they could edit it
they changed some of the synapses in it
and then they asked it where's the
Eiffel Tower and it said it's in Rome
and then they asked you know how do you
get there oh how do you get there from
Germany oh you take this train and to
Roma Termini train station and this and
that and what might you see if you're in
front of it oh you might see the
Coliseum so they had edit it so they
literally moved it to Rome but it the
way it's storing this information it's
incredibly dumb for for any fellow nerds
listening to this there was a big Matrix
and a
and roughly speaking there are certain
row and column vectors which encode
these things and the they correspond
very highly related principal components
and it will be much more efficient for
sparse Matrix just store it in the
database you know and and but and
everything so far we've figured out how
these things do our ways where you can
see they can easily be improved and the
fact that this particular architecture
has some roadblocks built into it is in
no way going to prevent um craft the
researchers from quickly finding
workarounds and making
other kinds of architectures
sort of go all the way so so it's um in
short it's turned out to be a lot
easier to build human close to human
intelligence than we thought then that
means our Runway is a species that
get our together has shortened
and it seems like the scary thing about
the effectiveness of large language
models uh so Sam Altman every
conversation with
and
he really showed that the leap from gpt3
to gpt4 has to do with just a bunch of
hacks
a bunch of
uh little Explorations but with the
smart researchers doing a few little
fixes here and there it's not some
fundamental leap and transformation in
the architecture and more data and more
compute and more data and compute but he
said the big leaps has to do with not
the data in the compute but just
learning this new discipline just like
you said so researchers are going to
look at these architectures and there
might be big leaps where you realize
wait why are we doing this in this dumb
way yeah and all of a sudden this model
is 10x smarter yeah and that that can
happen on any one day on anyone Tuesday
or Wednesday afternoon and then all of a
sudden you have a system that's 10x
smarter
um it seems like it's such a new
discipline it's such a new like we
understand so little about why this
thing works so damn well that uh the
linear Improvement of compute or
exponential but the steady Improvement
of compute steady Improvement of the
data may not be the thing that even
leads to the next leap it could be a
surprise little hack that improves
everything for a lot of little leaps
here and there because
because so much of this is out in the
open also
so many smart people are looking at this
and trying to figure out little leaps
here and there and uh it becomes this
sort of collective race where if a lot
of people feel if I don't take the leap
someone else with and this is actually
very crucial for for the other part but
why do we want to slow this down so
again what this open letter is calling
for is just pausing
all training
of uh
systems that are more powerful than gpt4
for six months
let's give a chance
for the labs to coordinate a bit on
safety and for society to adapt give the
right incentives to the labs because I
you know you've interviewed a lot of
these
people who lead these labs and you know
just as well as I do that they're good
people they're idealistic people they're
doing this
first and foremost because they believe
that AI has a huge potential to help
humanity and uh
but at the same time they are trapped in
this horrible race to the bottom
have you read meditations on malloc
by Scott Alexander yes yeah it's a
beautiful essay on this poem by Ginsburg
where he interprets it as being about
this monster
it's this game theory monster that that
pits people into against each other in
this they race the bottom where
everybody ultimately loses the edit the
evil thing about this monster is even
though everybody sees it and understands
they still can't get out of the race
right
most a good fraction of all the bad
things that we humans do are caused by
moloch and I I like uh Scott Alexander's
um
naming of the monster so we can we
humans can think of it as an f a thing
if you look at why do we have
overfishing why do we have more
generally the tragedy of the commons why
is it that um
I don't know if you've had her on your
podcast yeah she's become a friend yeah
great she made this awesome point
recently that beauty filters that a lot
of female
influencers feel pressured to use or
exactly malloc in action again first
nobody was using them
and people saw them just the way they
were and then some of them started using
it
and becoming ever more Plastic Fantastic
and then the other ones they weren't
using he started to realize that
if they want to just keep their
their market share they have to start
using it too
and that and then you're in a situation
where they're all using it
and and none of them has any more market
share or less than before so nobody
gained anything everybody lost
and they have to keep becoming ever more
Plastic Fantastic also right
and uh
but nobody can go back to the old way
because it's just
too costly right the malloc is
everywhere
and um
molok is not a new arrival on on the
scene either we humans have developed a
lot of collaboration mechanisms to help
us fight back against Malik through
various kinds of constructive
collaboration the Soviet Union and the
United States did sign the number of our
Arms Control treaties
against moloch who is trying to stoke
them into
unnecessarily risky nuclear arms races
Etc et cetera and this is exactly what's
happening on the AI front this time
it's a little bit geopolitics but it's
mostly money where there's just so much
commercial pressure you know if you take
any of these
leaders of the top tech companies
and if they just say you know this is
too risky I want to pause
for six months they're going to get a
lot of pressure from shareholders and
others
we're like well you know if you pause
but those guys don't pause
we're
if you don't want to get our lunch eaten
yeah and shareholders even have the
power to replace the the executives in
the worst case right so
we did this open letter because we want
to help these idealistic Tech Executives
to do
what their heart tells them by providing
enough public pressure on the whole
sector
just pause so they can all pause
in a coordinated fashion and I think
without the public pressure none of them
can do it alone
push back against their shareholders no
matter how good-hearted they are because
Malik is a really powerful foe
so the idea
is to
for the major developers of AI systems
like this so we're talking about
Microsoft Google
meta
and anyone else well open AI is very
close with Microsoft and there are
plenty of smaller players and throw for
example anthropic which is very
impressive there's conjecture there's
many many players I don't want to make a
long list so leave anyone out and
um
for that reason it's so important that
some coordination happens that there's
external pressure on all of them saying
you all need the Pawns because then the
the people the researchers in they were
these organizations the leaders who want
to slow down a little bit they can say
their shareholders you know
everybody's slowing down because of this
pressure and and it's the right thing to
do
have you seen in history their uh
examples what's possible to pause the
model absolutely
and even like human cloning for example
you could make so much money on human
cloning
why aren't we doing it
because biologists thought hard about
this like this is way too risky we they
got together well in the 70s in the
cinema and decided even
to stop a lot more stuff also just
editing the human germline right
Gene editing that goes in
to our offspring
and decided let's let's not do this
because it's too unpredictable what it's
going to lead to
we could lose control over what happens
to our species so they paused
uh there was a ton of money to be made
there so it's it's very doable but you
just need you need a public awareness of
the of what the risks are and the
broader Community coming in and saying
hey let's slow down and you know another
another common pushback I get today is
we We Can't Stop in the west because
China
and in China undoubtedly they also get
told we can't slow down because the West
because both sides think they're the
good guy yeah
but look at human cloning you know
did China Forge ahead with human cloning
there's been exactly one human cloning
that's actually been done that I know of
it was done by a Chinese guy do you know
where he is now right in jail
and you know who put him there
who Chinese government
not because westerners said China look
this is no the Chinese government put
them there because they also felt they
like control the Chinese government if
anything maybe they are even more
concerned about having control then the
Western governments have no incentive of
just losing control over where
everything is going
and you can also see the Ernie bot that
was released by I believe I do recently
they got a lot of pushback from the
government and had to rein it in you
know in a big way
um I think once this basic message comes
out that this isn't an arms race it's a
suicide race
where everybody loses if anybody's AI
goes out of control it really changes
the whole dynamic it it's not
it's
I'll say this again because this is this
very basic point I think a lot of people
get wrong because a lot of people
dismiss the whole idea that AI can
really get
very superhuman because they think
there's something really magical about
intelligence such that it can only exist
in human Minds you know because they
believe that they think it's going to
kind of get to just more or less
gpd4 plus plus and then that's it
they don't see it as a super as a
suicide race they think whoever gets
that first they're going to control the
world they're going to win
that's not how it's going to be and we
can talk again about
the the scientific arguments from why
it's not going to stop there but
the way it's going to be is if if
anybody completely loses control and you
know you don't care if if
some some if someone manages this
takeover the world who really doesn't
share your goals you probably don't
really even care very much about what
nationality they have you're not going
to like it it's much worse than today
uh who if it's if you live in orwellian
dystopia who you what do you care who's
created it right and if someone if it
goes farther and and
we just lose control even to the
machines
so that it's not US versus them it's US
versus it
what do you care who who created this
this underlying entity which has goals
different from humans ultimately and we
get marginalized we get made obsolete we
get replaced
that's why what I mean when I say it's a
suicide race you know it's um it's kind
of like we're rushing towards this cliff
but the closer to the cliff we get the
more Scenic the views are and the more
money there is there and the more so we
keep going
but we have to also stop at some point
right quit while we're ahead and uh
it's um
it's a suicide race which cannot be won
but the way that really benefit from it
is
to continue developing awesome AI a
little bit slower so we make it safe
make sure it does the things that humans
want and create a condition where
everybody wins the technology has shown
us that you know geopolitics and and
politics and general is not a zero-sum
game at all
so there is some rate of development
that will lead
us as a human species to lose control of
this thing and the hope you have is that
there's some lower level of development
which will not which will not allow us
to lose control this is an interesting
thought you have about losing control so
what if you have somebody if you're
somebody like Sandra pracha or Sam
Altman at the head of a company like
this you're saying if they develop an
AGI they too will lose control of it
so no one person can maintain control no
group of individuals can maintain if
it's if it's created very very soon and
as a big black box that we don't
understand like the large language
models yeah then I'm very confident
they're going to lose control but this
isn't just me saying you know Sam Altman
and then Mr sabis have both said
themselves acknowledge that you know
there's really great risks with this and
they they want to slow down once they
feel it gets scary
it's but it's clear that they're stuck
in this again molok is forcing them to
go a little faster than they're
comfortable with because of pressure
from just commercial pressures right
to get a bit optimistic here of course
this is a problem that can be ultimately
solved
uh
it's just to win this wisdom race
it's clear that what we hope that was
gonna happen hasn't happened the the
capability progress has gone faster than
a lot of people thought then and the
part the progress in in the public
sphere of policy making and so on has
gone slower than we thought even the
technical AI safety has gone slower a
lot of the technical Safety Research was
kind of banking on that um
large language models and other poorly
understood systems couldn't get us all
the way that you had to build more of a
kind of intelligence that you could
understand maybe it could prove itself
safe you know things like this
and um
I'm quite confident that this can be
done um so we can reap all the benefits
but we cannot do it as quickly as uh
this is out of control Express train
we're on now is gonna get the AGI that's
why we need a little more time I feel
is there something to be said well like
Sam Allman talked about which is while
we're in the pre-agi stage to release
often and as transparently as possible
to learn a lot
so as opposed to being extremely
cautious release a lot don't uh don't
invest in a closed development where you
focus on AI safety while is somewhat
dumb
quote unquote
uh release as often as possible and as
you start to see signs of
uh human level intelligence or
superhuman level intelligence then you
put a halt on it well
what a lot of safety researchers have
been saying for many years is the most
dangerous things you can do with an AI
is first of all teach it to write code
yeah because that's the first step
towards recursive self-improvement which
can take it from AGI to much higher
levels okay oops we've done that
and uh another thing high risk is
connected to the internet Let It Go to
websites download stuff on its own and
talk to people
oops we've done that already you know
Elias yukowski you said you interviewed
him recently right yeah so he had this
tweet recently which said
gave me one of the best laughs in a
while and he's like hey people used to
make fun of me and say you're so stupid
Eliezer because you're saying you're
saying um
you have to worry of obviously
developers wants to get to like really
strong AI first thing you're going to do
is like never connect it to the internet
Keep It In The Box yeah where you know
where you can really study it
so he had written it in the like in the
meme form so it's like then yeah and
then that and then now
let's LOL let's make a chatbot
yeah yeah and the third thing is Stuart
Russell yeah you know
amazing AI researcher he had he has
argued for a while that
we should never teach AI anything about
humans
above all we should never let it learn
about human psychology and how you
manipulate humans
that's the most dangerous kind of
knowledge you can give it yeah you can
teach it all it needs to know how to
about how to cure cancer and stuff like
that but don't let it read Daniel
kahneman's book about cognitive biases
and all that and then
oops lol you know let's invent social
media
I'll recommender algorithms which do
exactly that they they get so good at
knowing us and pressing our buttons
that we've we're starting to create a
world now where we just have ever more
hatred
because they figured out that these
algorithms not for out of evil but just
to make money on Advertising that the
best way to get more engagement
the euphemism
get people glued to their little
rectangles right is just to make them
pissed off that's really interesting
that a large AI system that's doing the
recommender system kind of task on
social media is basically just studying
human beings because it's a bunch of us
rats giving it signal
non-stop signal it'll show a thing and
it would give signal on whether we
spread that thing we like that thing
that thing increases our engagement gets
us to return to the platform and it has
that on the scale of hundreds of
millions of people constantly so it's
just learning and learning and learning
and presumably if the param the number
of parameters the neural network that's
doing the learning and more end-to-end
the learning is
the more it's able is just to basically
encode how to manipulate human behavior
how to control humans at scale exactly
and that is not something you think is a
new man in his interest
yes right now it's mainly letting some
humans manipulate other humans for
profit
and Power
which is already
caused a lot of damage and eventually
that's a sort of
skill that can make ai's persuade humans
to let them escape and whatever safety
precautions yeah but you know there was
a really nice article um and the New
York Times recently by a you all know a
Harari and and um two co-authors
including Justin Harris from the social
dilemma and
they have this phrase in there I love
Humanity's first contact with Advanced
AI
or social media
and we lost that one
we now live in a country where there's
much more hate in the world where
there's much more hate in fact
and in our democracy that we're having
this conversation then people can't even
agree on who won the last election you
know
and we humans often point fingers at
other humans and say it's their fault
but it's really molok and these AI
algorithms
we got the algorithms and then molok
pitted the social media companies around
against each other so nobody could have
a less creepy algorithm because then
they would lose out on our Revenue to
the other company is there any way to
win that battle back just if we just
Linger on this one battle that we've
lost in terms of social media is it
possible
to redesign social media this very
medium in which we use as a civilization
to communicate with each other to have
these kinds of conversations to have
discourse to try to figure out how to
solve the biggest problems in the world
whether that's nuclear war or the
development of AGI is is it possible
uh to do social media correct I think
it's not only possible but it's it's
necessary who are we kidding that we're
going to be able to solve all these
other challenges if we can't even have a
conversation with each other that's
constructive the whole idea the key idea
of democracy is that you get a bunch of
people together
and they have a real conversation the
ones you try to Foster on this podcast
or you respectfully listen to people you
disagree with
and you realize actually you know there
are some things actually we some common
ground we have and that's it's yeah we
both agree let's not have a nuclear Wars
let's not do that
um etc etc
we're kidding ourselves the thinking we
can face the off the
second contact with with ever more
powerful AI that's happening now with
this large language models if we can't
even
have a functional
conversation in the public space that's
why I started to improve the news
project to improve the news.org but um
I I'm an optimist fundamentally in um
and that there is a lot of intrinsic
goodness in in in people
and that uh what makes the difference
between someone doing good things for
for Humanity and bad things is not
some sort of fairy tale thing that this
person was born with the evil Gene and
this one was not born with a good Gene
no I think it's whether we put whether
people
find themselves in situations that bring
out the best in them or they bring out
the worst in them and I feel we're
building an internet
and a society that brings out the worst
but it doesn't have to be that way no it
does not it's possible to create
incentives and also create incentives
that make money they both make money and
bring out the best in people I mean in
the long term it's not a good investment
for anyone you know to have a nuclear
war for example
and you know is it a good investment for
Humanity if we just ultimately replace
all humans by machines and then we're so
obsolete that eventually the
there are no humans left
well it depends against how you do the
math but like if I would say by any
reasonable economic started if you look
at the future income of humans and there
aren't any you know that's not a good
investment
moreover like why why can't we have a
little bit of pride in our species damn
it you know why should we just build
another species that gets rid of us if
we were Neanderthals
would we really consider it a smart move
if the
if we had really Advanced biotech to
build homo sapiens
you you know you might say hey Max you
know yeah let's build build the
these Homo sapiens they're going to be
smarter than us maybe they can help us
defend us better against the Predators
and help fix their bar caves make them
nicer and we'll control them undoubtedly
you know so then they build build a
couple a little baby girl little baby
boy you know and
and then you have some some wise old and
Neanderthal Elder was like hmm I'm
scared that uh we're opening in
Pandora's Box here and that we're going
to get outsmarted by these
super Neanderthal intelligences and
there won't be any neanderthals left and
then but then you have a bunch of others
in the cave right yeah are you such a
Luddite scaremonger of course they're
going to want to keep us around because
we are their creators and and why you
know the smaller I think the smarter
they get the nicer they're gonna get
they're gonna leave us they're gonna
they're going to want this around and
it's going to be fine and and besides
look at these babies they're so cute
clearly they're totally harmless that's
exact those babies are exactly gpt4 yeah
it's not I want to be clear it's not
gpt4
that's terrifying it's the gpt4 is a
baby technology
you know and Microsoft even had a paper
recently out
uh with a title something like sparkles
of AGI whatever basically saying this is
baby AI
I like these little Neanderthal babies
and it's going to grow up there's going
to be other systems from from the same
company from other companies they'll be
way more powerful and but they're going
to take all the things
ideas from these babies
and before we know it we're gonna be
like
those last neanderthals who are pretty
disappointed and when they realized that
they were getting replaced well this
interesting point you make which is the
programming is it's entirely possible
that gpt4 is already
the kind of system that can
change everything
by writing programs sorry it's yeah it's
because it's Life 2.0
the systems I'm afraid of are going to
look nothing like a large language model
and they're not
but once it gets once it or other people
figure out a way of using this Tech to
make much better Tech right it's just
constantly replacing its software and
from everything we've seen about how how
these work under the hood they're like
the minimum viable intelligence they do
everything you know the dumbest way that
still works sort of yeah and um
so they were life 3.0 except when they
replace their software it's a lot faster
than when you when when you decide to
learn Swedish
and moreover they think a lot faster
than us too so when uh you know we don't
think uh have
one
logical step every nanosecond or a few
or so the way they do and we can't also
just suddenly scale up our Hardware
massively in the cloud which we're so
limited right
so they are in it they are also life
have
can soon be become a little bit more
like life 3.0 and that if they need more
Hardware hey just rent it in the cloud
you know how do you pay for it well with
all the services you provide
yeah
and what we haven't seen yet
which could change a lot is uh entire
Software System so right now programming
is done sort of in bits and pieces uh as
as an assistant tool to humans but I do
a lot of programming and with the kind
of stuff that gbt4 is able to do I mean
it's replacing a lot what I'm able to do
but I you still need a human in the loop
to kind of manage the design of things
manage like what are the prompts that
generate the kind of stuff to do some
basic adjustment of the codes let's do
some debugging but if it's possible to
add on top of GPT for kind of a feedback
loop of of uh self-debugging improving
the code and then you launch that system
out into the wild on the internet
because everything is connected and have
it do things have it interact with
humans and then get that feedback now
you have this giant ecosystem yeah of
humans that's one of the things that uh
yeah Elon Musk recently sort of tweeted
as a case why everyone needs to pay
Seven dollars or whatever for Twitter to
make sure they're real they're make sure
they're real we're now going to be
living in a world where the the Bots are
getting smarter and smarter and smarter
to a degree where you can't uh you can't
tell the difference between a human and
a bot that's right and now you can have
uh Bots outnumber humans by yeah one
million to one which is why he's making
the case why you have to pay yeah to
prove you're human which is one of the
only mechanisms which is depressing and
I yeah I feel we have to remember
as individuals we should from time to
time ask ourselves why are we doing what
we're doing all right then as a species
we need to do that too
so if we're building as as you say
machines that are outnumbering us
and more and more outsmarting us and and
replacing us on the job market not just
for the dangerous and and boring tasks
but also for writing poems and doing art
and things that a lot of people find
really meaningful God ask yourself why
why are we doing this
uh we are the answer is moloch is
tricking us into doing it
and it's such a clever trick that even
though we see the trick we still have no
choice but to fall for it right
come also the thing you said about you
using uh
co-pilot AI tools to program faster how
many time what factor faster would you
say your code now does it go twice as
fast or I don't really
uh because it's such a new tool yeah
it's I don't know if speed is
significantly improved but it feels like
I'm a year away from being
uh five to ten times faster so if that's
typical for programmers then uh you're
already seeing another kind of self
recursive self-improvement right because
previously uh one like a major
generation of improvement of the codes
would happen on the human r d time scale
and now if that's five times shorter
then it's going to take five times less
time than it otherwise would to develop
the next level of these tools and so on
so this these These are the this is
exactly the sort of beginning of an of
an intelligence explosion they can be
humans in the loop a lot in the early
stages and then eventually humans are
needed less and less and the machines
can more kind of go along but you what
you said there is just an exact example
of these sort of things another thing
which which um
I was kind of lying on my psychiatrist
imagining I'm on a psychiatrist's couch
here saying what are my fears that
people would do with um AI systems
another
so I mentioned three that I had fears
about many years ago that they would do
uh namely uh teacher the code uh
connected to the internet then teach it
to manipulate humans a fourth one is
building an API
where code can control all the super
powerful thing right that is very
unfortunate because one thing that
systems like gpt4 have going for them is
that they are an Oracle in the sense
that they just answer questions there's
no robot connected to the gpt4 gpt4
can't go and do stock trading based on
its thinking yeah it is not an agent an
intelligent agent is something that
takes in information from the world
processes it
to figure out what action to take based
on its goals that it has and then does
something
back on the world but once you have an
API for example tpd4 Nothing Stops Joe
schmoe and all and a lot of other people
from building real agents
which just keep making calls somewhere
in some inner loop somewhere to these
powerful Oracle systems
and which makes them themselves much
more powerful that's another kind of um
unfortunate development which I think we
would have been better off uh delaying I
don't want to pick on any particular
companies I think they're all under a
lot of pressure to make money yeah
and um
again we the reason we're calling for
this pause is to give them all cover to
do what they know is the right thing
slow down a little bit at this point but
everything we've talked about I hope
will can will make it clear that people
watching this you know why
these sort of human level tools can
cause a gradual acceleration you keep
using yesterday's technology to build
tomorrow's technology yeah and when you
do
that over and over again you naturally
get an explosion you know that's the
definition of an explosion in science
right like
if you have
two people
um they fall in love now you have four
people and then they can make more
babies and now you have eight people and
then then you have 16 32 64 Etc that's
we call that a population explosion
where it's just that each
if it's instead free neutrons in a
nuclear reaction that if H1 can make
more than one then you get an
exponential growth in that we call it a
nuclear explosion all explosions are
like that and an intelligence explosion
it's just exactly the same principle
that some quantity some amount of
intelligence can make more intelligence
than that and then repeat you always get
exponentials
what's your intuition why does you
mention there's some technical reasons
why it doesn't stop at a certain point
what's your intuition and uh do you have
any intuition why it might stop
it's obviously going to stop when it
bumps up against the laws of physics
there are some things you just can't do
no matter how smart you are right
allegedly
um
laws of physics yeah yeah right Seth
Lloyd wrote a really cool paper on the
physical limits on uh computation for
example if you make it
put too much energy into it and finite
space it'll turn into a black hole you
can't move information around Fashion
the speed of light stuff like that but
uh it's hard to store
way more than than a modest number bits
per atom Etc but you know those limits
are just astronomically above like 30
orders of magnitude above where we are
now so no
bigger different bigger jump in
intelligence than if you go from uh from
an ant to a human
I think
of course what we want to do is have
have a controlled
thing the nuclear reactor you put
moderators in to make sure exactly it
doesn't blow up out of control right
when we do um
experiments with Biology and cells and
so on you know we also try to make sure
it doesn't get out of control
um
we can do this with ai2 the thing is we
haven't succeeded yet and Morlock is
exactly
doing the opposite just fueling just
egging everybody on faster faster faster
or the other company is going to catch
up with you or the other country is
going to catch up with you
really those we have to want this stuff
we have and and I don't believe in this
just asking people to look into their
hearts and do the right thing it's
easier for others to say that but like
if if you're in a situation where your
company is going to get screwed
if you
by other companies that are not stopping
you know you're putting people in a very
hard situation the the right thing to do
is change the whole incentive structure
instead
and and this is not an old
maybe I should say one more thing about
this because molok has been around as
Humanity's
number one or number two enemy since the
beginning of civilization and we came up
with some really cool counter measures
like first of all already over a hundred
thousand years ago Evolution realized
that it was very unhelpful that people
kept killing each other all the time
yeah so it genetically
gave us compassion
and made it so that it like if you get
two drunk dudes getting into a pointless
bar fight
they might give each other black eyes
but they have a lot of inhibition
towards towards just killing each other
that's a gen and similarly if you find a
baby lying on the street when you go out
for your morning jog tomorrow you're
gonna stop and pick it up right even
though maybe it make you late for your
next podcast
so Evolution gave us these genes that
make our own egoistic incentives more
aligned with what's good for the greater
group or part of right and then uh as we
got a bit more sophisticated and
developed language
we invented gossip which is also a
fantastic anti-malark right because now
it it really discourages
Liars Moochers cheaters because it their
own incentive now is not to do this
because word quickly gets around and
then suddenly people aren't going to
invite them to their dinners anymore or
trust them and then when we got still
more sophisticated and bigger societies
you know invented the legal system
where even strangers who didn't couldn't
rely on gossip and and things like this
would treat each other would have an
incentive now those guys in the bar
fights even if they someone is so drunk
that he
actually wants to kill the other guy
he also has a little thought on the back
of his head that you know do I really
want to spend the next 10 years
eating like really crappy food in a
small room uh I'm just gonna I'm just
gonna chill out you know so and we we
similarly have tried to give these
incentives to our corporations by having
having regulation and all sorts of
oversight so that their incentives are
aligned with the greater good we've
tried really hard
um and um the the big problem that we're
failing now
is not that we haven't tried before but
it's just that the tech is growing much
is developing much faster than The
Regulators been able to keep up right so
Regulators it's kind of comical the
European Union right now is doing this
AI Act Right which and in the beginning
they had a little opt-out exception that
gpt4 would be completely excluded from
regulation
brilliant idea what's the logic behind
that
some lobbyists post successfully for
this so we were actually quite involved
with the future Life Institute
um Mark Bracco Mr ook Anthony gear and
others you know we're quite involved
with um talking to very educating
various people involved in this process
about
these general purpose AI models coming
and pointing out that they would become
the laughing stock if they didn't put it
in so it the French started pushing for
it it got put in to the draft and it
looked like all was good and then there
was a huge counter push from lobbyists
yeah there were more lobbyists in
Brussels from tech companies and from
oil companies for example and it looked
like it might it's going to maybe get
taken out again
and now gpt4 happened
and I think it's going to stay in but
this just shows you know monologue can
be defeated
but the the challenge we're facing is
that the tech is generally much faster
than what the
policymakers are
and a lot of the policy makers also
don't have a tech background so it's you
know we really need to work hard to
educate them on on how on what's taking
place here
so so we're getting the situation where
the first kind of non so you know I
Define artificial intelligence just as
non-biological intelligence right
and by that definition
a company a corporation is also an
artificial intelligence because the
corporation isn't it's humans it's the
system
if its CEO decides the CEO of a tobacco
company decides one morning that she or
he doesn't want to sell cigarettes
anymore they'll just put another CEO in
there
it's not enough to align the incentives
of individual people or in align
individual computers incentives
to their owners which is what
technically AI Safety Research is about
you also have to align the incentives of
Corporations
with a greater good and some
corporations have gotten so big and so
powerful very quickly and that in many
cases they're lobbyists instead align
The Regulators to what they want rather
than the other way around the classic
regulatory capture right is is uh the
thing that the Slowdown hopes to achieve
is give enough time to the regulars to
catch up or enough time to the companies
themselves to breathe and understand how
to do AI safety correctly I think both
and but I think that the vision the path
to success I see is first you give a
breather actually to to the people in
these companies their leadership who
wants to do the right thing and they all
have safety teams and so on other
companies give them a chance to
get together with the other companies
and the outside pressure can also help
catalyze that right
and and work out
what is it that's
what are the reasonable safety
requirements one should put on future
systems before they get rolled out
there are a lot of people also in
Academia and elsewhere outside of these
companies who can be brought into this
and have a lot of very good ideas
and then um
I think it's very realistic that within
six months
you can get these people coming up so
here's a white paper here's where we all
think it's reasonable
um you know you didn't just because cars
killed a lot of people they didn't ban
cars but they got together a bunch of
people and decided you know in order to
be allowed to sell a car it has to have
a seat belt in them
they're the analogous things that you
can start requiring a future AI systems
so that they are are safe
and uh once this have this heavy lifting
this intellectual work has been done by
experts in the field which can be done
quickly
I think it's beginning to be quite easy
to get policy makers to
to see yeah this is a good idea and it's
it's you know
for the fight for the companies to fight
molok
they want and I I believe Sam Altman has
explicitly called for this they want The
Regulators to actually adopt it so that
their competition is going to abide by
it too right you don't want
uh
you don't want to be invade enacting all
these principles then you abide by them
and then there's this one little
company that
doesn't sign on to it and then now they
can gradually overtake you then the
companies will get
be able to sleep it's secure knowing
that everybody's playing by the same
rules so do you think it's possible to
develop guard rails
that keep the systems from
uh for basically damaging irreparably
Humanity while still enabling sort of
the capitalist fueled competition
between companies as they develop how to
best make money with this AI you think
there's a balancing totally that's
possible absolutely I mean we've seen
that in many other sectors where you've
had the free market produce quite good
things without uh causing particular
harm
um when the guardrails are there and
they work
you know
capitalism is very effective good way of
optimizing for just getting the same
things done more efficiently it was but
it was good you know and like in
hindsight and I never met anyone
even
on parties way over on the right in in
any country who think it was a bad
thinks it was a terrible idea to ban
child labor for example so
yeah but it seems like this particular
technology has gotten so good so fast
become
powerful
to a degree where you could see in the
near term the ability to make a lot of
money and to put guard rails develop
guard rails quickly in that kind of
contact seems to be tricky it's not uh
similar to cars or child labor it seems
like the opportunity to make a lot of
money here very quickly is right here
before so again there's this cliff
on the ground you can pick up or
whatever so you want to drive there very
fast but it's not in anyone's incentive
that we go over the cliff and it's not
like everybody's in their own car all
the cars are connected together with a
chain yeah so if anyone goes over
they'll start dragging others down the
other's down too and so ultimately it's
in the selfish
interests also of the people in the
companies to to
slow down when the when it just start
seeing the characters of The Cliff there
in front of you right and the problem is
that um even though the people who are
building the technology
and the CEOs they really get it the
shareholders and these other Market
forces they are people who don't
honestly
understand that the cliff is there they
usually don't you have to get quite into
the weeds to really appreciate how
powerful this is and how fast and a lot
of people are even still stuck again in
this idea that
intelligence in this carbon chauvinism
as I like to call that that you can only
have our level of intelligence in
humans that there's something magical
about it whereas the people in the tech
companies
who build this stuff they all realize
that intelligence is
information processing of a certain kind
and it really doesn't matter at all
whether the information is processed by
carbon atoms in neurons in brains or by
silicon atoms and some technology we
build
you brought up capitalism earlier and
there are a lot of people who love
capitalism and a lot of people
who really really
don't and
it struck me recently that the what's
happening with capitalism here is
exactly analogous to the way in which
Super intelligence might wipe us out
so
you know you know I studied economics
for my undergrad Stockholm School of
Economics yay
well no no I tell me someone's very
interested in how how you could use
Market forces to just get stuff done
more efficiently but give the right
incentives to Market so that it wouldn't
do really bad things
so Dylan had filmonell who's a a
professor and colleague of mine at MIT
wrote this really interesting paper with
some collaborators recently
where they proved mathematically that if
you just up take one goal that you just
optimized for
on and on and on indefinitely
do you think it's gonna
bring you in the right direction
what basically always happens is in the
beginning it will make things better for
you
but if you keep going at some point
that's going to start making things
worse for you again and then gradually
it's going to make it really really
terrible so just as a simple
the way I think of the proof is like
suppose you want to go from here
back to Austin for example and you're
like okay yeah let's just let's go south
but you put in exactly the right sort of
the right direction just optimize that
South as possible you get closer and
closer to Austin
but uh you there's always some little
error So you you're not going exactly
towards Austin but you get pretty close
but eventually you start going away
again and eventually you're going to be
leaving the solar system yeah and they
they proved it's a beautiful
mathematical proof this happens
generally
and this is very important for AI
because for even though Stuart Russell
has
written a book and given a lot of talks
on why it's a bad idea to have ai just
blindly optimize something that's what
pretty much all our systems do yeah we
have something called the loss function
that we're just minimizing or reward
function we just minimize maximizing
stuff
and um
capitalism is exactly like that too we
want we wanted to get stuff done more
efficiently that people wanted so
introduce the free market
things got done much more efficiently
than they did in
say communism right and it got better
but then it just kept optimizing it and
kept optimizing and you got every bigger
companies and every more efficient
information processing and now also very
much powered by it
and uh eventually a lot of people are
beginning to feel wait we're kind of
optimizing a bit too much like why did
we just chop down half the rainforest
you know and why why did suddenly these
Regulators get
captured by lobbyists and so on it's
just the same optimization that's been
running for too long
if you have an AI that actually has
power over the world and you just give
it one goal and just like keep
optimizing that most likely everybody's
gonna be like Yay this is great in the
beginning things are getting better
but um
it's almost impossible to give it
exactly the right direction to optimize
in and then
eventually all hey Breaks Loose right
Nick Bostrom and others are given the
example to sound quite silly like what
if you just want to like tell it
cure cancer or something and that's all
you tell it maybe it's going to decide
to
take over entire continents just so we
can get more super computer facilities
in there and
figure out how to cure cancer backwards
and then you're like wait that's not
what I want then right and
um
the the the issue with capitalism and
the issue with runaway I have kind of
merged now because the malloc I talked
about
is exactly the capitalist molloch that
we have built an economy that has
optimizing for only one thing
profit
right and that worked great back when
things were very inefficient and then
now it's getting done better and it
worked great as long as the companies
were small enough that they couldn't
capture the regulators
but
that's not true anymore but they keep
optimizing
and now
they realize that that they can these
companies can make even more profit by
building ever more powerful AI even if
it's Reckless
but optimize more and more and more and
more and more
so this is molok again showing up and I
just want to anyone here who has any
concerns about about uh late stage
capitalism having gone a little too far
you should worry about
super intelligence because it's the same
villain in both cases it's more like and
optimizing one objective function
aggressively blindly is going to take us
there yeah we have this pause from time
to time and look into our hearts and ask
why are we doing this is this uh am I
still going towards Austin or have I
gone too far you know maybe we should
change direction
and that is the idea behind the halt for
six months what six months it seems like
a very short period just can we just
linger and explore different ideas here
because this feels like a really
important moment in human history where
pausing would actually have a
significant positive effect
we said six months because we figured
the number one pushback that we're gonna
get in the west was like but China
and
everybody knows there's no way that
China is going to catch up with the West
on this in six months so it's that
argument goes off the table and you can
forget about geopolitical competition
and just focus on
the real issue that's why we put this
that's really interesting but you've
already made the case that uh even for
China if you actually want to take on
that argument China too would not be
bothered by a longer halt because they
don't want to lose control even more
than the West doesn't
that's what I think that's a really
interesting argument like I have to
actually really think about that which
the the kind of thing people assume is
if you develop an AGI
that open AI if they're the ones that do
it for example they're going to win but
you're saying no they're everybody loses
yeah it's gonna get better and better
and better and then Kaboom we all lose
that's what's gonna happen when lose and
win are defined in a metric of basically
quality of life
for human civilization and for Sam
Altman
to be blunt my personal guess you know
and people can quibble with this is that
we're just gonna they won't be any
humans that's it that's what I mean by
lose
you know if you if we can see in history
once you have some species or some group
of people who aren't needed anymore
it doesn't usually work out so well for
them right yeah
there were a lot of horses for that were
used for traffic in Boston and then the
car got invented and most of them got
you know well we don't need to go there
and uh if you look at them
humans you know right now we why did the
labor movement succeed and after the
Industrial Revolution because it was
needed
even though we had a lot of mallocs and
there was child labor and so on you know
the company still needed to have workers
and that's why strikes had power and so
on if we get to the point where most
humans aren't needed anymore I think
it's like it's quite naive to think that
they're going to still be treated well
you know we say that yeah yeah um
everybody's equal and the government
will always will always protect them but
if you look in practice
groups that are very disenfranchised and
don't have any actual power
it usually gets screwed
and uh now in in the beginning so
Industrial Revolution
we automated away muscle work
but that got went worked out pretty well
eventually because we educated ourselves
and started doing working with our
brains instead and got usually more
interesting better paid jobs
but now we're beginning to replace brain
work so we've replaced a lot of boring
stuff like we got the pocket calculator
so you don't have people adding
multiplying numbers anymore at work fine
they were better jobs they could get but
now dpt4 you know
and the stable diffusion and techniques
like this they're really beginning to
blow away some real some jobs that
people really loved having it was a
heartbreaking article just post just
yesterday on social media I saw about
this guy who was doing 3D modeling for
gaming and if
and all of a sudden now they got this
new software you just give says prompts
and he feels his whole job that he loved
it's lost its meaning you know and uh
I asked the gpt4 to rewrite Twinkle
Twinkle Little Star in the style of
Shakespeare
I couldn't have done such a good job it
was just really impressive you've seen a
lot of the Arts coming out here right so
I'm all for
automating away the dangerous jobs and
the boring jobs but I think um
you hear some arguments which are too
glib sometimes people say well that's
all that's going to happen we're getting
rid of the boring boring uh tedious
dangerous jobs it's just not true there
are a lot of really interesting jobs
that are being taken away now journalism
is getting going to get crushed uh
coding is gonna get crushed like I
predict uh the job market for
programmers salaries are going to start
dropping
you know if you said you can code five
times faster you know then you need five
times fewer programmers maybe there will
be
more output also but you'll still end up
using fewer programs needing fewer
programmers than today and I love coding
you know I I think it's super cool
um so we we need to stop and ask
ourselves why again are we doing this as
humans like
I feel that AI should be built by
Humanity for Humanity and let's not
forget that it shouldn't be by malloc
for molok or what it really is now is
kind of by Humanity for moloch which
doesn't make any sense it's for us that
we're doing it then and um
it would make a lot more sense if we
build the develop figure out gradually
safely how to make all this Tech and
then we think about what are the kind of
jobs that people really don't want to
have you know and automate them all the
way and then we ask what are the jobs
that
people really find meaning in like maybe
taking care of children in the daycare
center maybe doing art etc etc and and
even if it were possible to automate
that away but we don't need to do that
right we built these machines
well it's possible that we
redefine or ReDiscover what are the jobs
that give us meaning so for me the thing
it is really sad like I
half the time I'm excited half the time
I'm uh crying as I'm yes I'm generating
code because
I kind of love programming it's uh it's
the Act of Creation you you have an idea
you design it and then you bring it to
life and it does something especially if
there's some intelligence that it does
something it doesn't even have to have
intelligence printing hello world on
screen you you you made a little machine
and it comes to life yeah and uh there's
a bunch of tricks you learn along the
way because you've been doing it for for
many many years and then for to see AI
be able to generate all the tricks you
thought were special yeah
um I don't know it's very
it
um it's it's scary it's almost painful
like a loss uh loss of Innocence maybe
like yeah maybe when when I was younger
uh I remember before I learned that
sugar is bad for you you should be on a
diet
I remember I enjoyed candy deeply in a
way I just can't anymore that I know is
bad for me I enjoyed it unapologetically
fully just intensely and I just I lost
that now I feel like a little bit of
that is lost for me with program or
being lost with programming similar as
it is for uh the the 3D modeler no
longer being able to really enjoy the
art of modeling uh 3D things for gaming
I don't know I don't know what to make
sense of that maybe I would ReDiscover
that the true magic of what it means to
be human is connecting with other humans
to have conversations like this I don't
know to uh to have sex to have to eat
food to really intensify the value from
conscious experiences versus like
creating other stuff you're pitching the
rebranding again from Homo sapiens
but the meaningful experiences and just
to inject some optimism in this here so
we don't sound like a bunch of gloomers
you know we can totally have our cake
and eat it you hear a lot of totally
claims that we can't afford
having more teachers yeah you have to
cut the number of nurses you know that's
just nonsense obviously
with anything even
quite far short of AGI we can
dramatically improve
grow the GDP and produce this wealth of
goods and services it's very easy to
create a world where everybody is better
off than today
including the richest people can be
better off as well right it's not a
zero-sum game you know technology again
you can have two countries like Sweden
and Denmark had all these ridiculous
Wars Century after century
and uh
sometimes that Sweden got a little
better off because it got a little bit
bigger and then Denmark got a little bit
better off because being a little bit
smaller and and but then we then
technology came along and we both got
just dramatically wealthier without
taking away from anyone else it was just
a total win for everyone
and uh AI can do that on steroids
if you can build safe AGI
if you can build super intelligence yeah
basically all the limitations that cause
harm today can be completely can be
completely eliminated right it's a
wonderful you talk possibility in this
this is not sci-fi this is something
which is clearly possible according to
laws of physics
and I we can talk about ways of making
it safe also
um but unfortunately
that'll only happen if we stare in that
direction that's absolutely not the
default outcome that why
income inequality keeps going up that's
why the life expectancy in the US has
been going down now I think it's four
years in a row I just read a
heartbreaking study from CDC about how
something like one-third of all teenage
girls in the US have been thinking about
suicide you know like
those are steps and they're totally the
wrong direction and and it's important
to keep our eyes on the prize here that
we can we have the power now for the
first time in the history of our species
to harness artificial intelligence to
help us really flourish
and help bring out the best in our
Humanity rather than the worst of it
to help us have really fulfilling
experiences that feel truly meaningful
and you and I shouldn't sit here and
dictate the future Generations what they
will be let them figure it out but let's
give them a chance to live and and not
foreclose all these possibilities for
them by just messing things up right
well for that we'll have to solve the AI
safety problem I just it would be nice
if we can link on exploring that a
little bit so one interesting way to
enter that discussion is uh you tweeted
and Elon replied you tweeted let's not
just focus on whether gpt4 we'll do more
harm or good on the job market but also
whether it's coding skills will hasten
the arrival of super intelligence that's
something we've been talking about right
so Elon proposed one thing in the reply
saying maximum truth seeking is my best
guess for AI safety can you maybe uh
Steel Man the case for this uh since
this objective function of Truth and uh
maybe make an argument against it and in
general what uh are your different ideas
to start approaching the solution to AI
safety I didn't see that reply actually
oh interesting all right so but I really
resonate with it because
AI is not evil it it caused people
around the world to hate each other much
more
but that's because we made it in a
certain way it's a tool we can use it
for great things and bad things and we
could just as well have ai systems and
this is this is part of my vision for
Success here
truth seeking AI that really brings us
together again
you know why do people hate each other
so much between countries and within
countries it's because
these have totally different versions of
the truth right
if they all have the same truth that
they trusted for good reason because
they could check it and verify it and
not have to believe in some
self-proclaimed Authority right
they wouldn't be as nearly as much hate
there'd be a lot more understanding
instead and uh
this is
I think something AI can help enormously
with
for example
a little baby step in this direction is
this website called metaculous where
People BET and make predictions not for
money but just for their own reputation
and it's kind of funny actually you
treat the humans like you treat AIS you
have a loss function where they get
penalized if they're super confident on
something and then the opposite happens
yeah
whereas if you're kind of humble and
then you're like I think it's 51 chance
this is going to happen and then the
other happens you don't get penalized
much and
and what you can see is that some people
are much better at predicting than
others they've earned your trusts right
one project that I'm working on right
now is the outgrowth to improve the news
Foundation together with the metaculous
focuses seeing if we can really scale
this up a lot with more powerful AI
because I would love it
I would love for there to be like a
really powerful truth seeking system
where
that is
trustworthy because it keeps being right
about stuff
and people who come to it and
maybe look at its latest trust ranking
of different pundits and newspapers Etc
if they want to know why some someone
got a low score they can click on it and
see all that predictions that they
actually made and how they turned out
you know
this is how we do it in science you
trust scientists like Einstein who said
something everybody thought was
and turned out to be right
you get a lot of trust Point than he did
it multiple times even
I think AI has the power to really heal
a lot of the the Rifts we're seeing by
creating Trust Systems
it has to get away from this idea today
with some fact-checking sites which
might themselves have an agenda and you
just trust it because of its reputation
to
you want to have it so this so these
sort of systems they are in their trust
and that's completely transparent
this I think would actually help a lot
uh that you know I think help heal the
very dysfunctional conversation that
Humanity has about how it's going to
deal with all its
biggest challenges in in the world today
and then uh
enter on the technical side you know
another common sort of Gloom uh
comment I get from people are saying
we're just screwed there's no hope
is well things like gpt4 are way too
complicated for a human to ever
understand and prove that they can be
trustworthy they're forgetting that AI
can help us prove that things work right
yeah and and there's this very
fundamental fact that in math it's much
harder to come up with a proof that it
is to verify that the proof is correct
you can actually write a little proof
checking code it's quite short
but you can assume and understand and
then it can check the most monstrously
long proof ever Drake generated even by
a computer and say yeah this is valid
so so right now
we we have
this uh
this approach with virus checking
software that it looks to see if there's
something you should not trust it and if
it can prove to itself that you should
not trust that code it warns you right
what if you flip this around
and this is an idea I should give credit
to Steve I'm 100 for so that it will
only run the code if it can prove
instead of not running it if it can
prove that it's not trustworthy if it
will only run and if it can prove that
it's trustworthy so it asks the code it
prove to me that you're going to do what
you say you're going to do
and and it gives you this proof
and you a little proof that you can
check it now you can actually trust
an AI That's much more intelligent than
you are right
because you it's it's problem to come up
with this proof that you could never
have found that you should trust it so
this is the interesting point I agree
with you but this is where Eleazar
yakowski might disagree with you his
claim not with you but with this idea
is his claim is super intelligent AI
would be able to know how to lie to you
with such a proof
I'll Delight to you and give me a proof
that I'm gonna think is correct yeah so
but it's not me it's lying to that's the
trick my proof checker
which is a piece of code so his general
idea is a super intelligent system
can lie to a dumber proof checker
so you're going to have as a system
becomes more and more intelligent
there's going to be a threshold
or a super intelligent system will be
able to effectively lie to a slightly
Dumber AGI system uh like there's a
threat like he really focuses on this
weak AGI the strong AGI jump with a
strong AGI can make all the weak age you
guys think that it's just one of them
but it's no longer that
and that leap is when it runs away yeah
I I don't buy that argument I think no
matter how super intelligent an AI is
it's never going to be able to prove to
me that they're only finitely many
primes for example
it can try to snow Me by making up all
sorts of the new weird rules of
deduction
that and say trust me you know
the way your proof jacket work is too
limited and we have this new hyper map
and it's true
um but then I would I would just uh take
the attitude okay I'm gonna forfeit some
of these this supposedly super cool
technology so I'm only going to go with
the ones that I can prove in my own
trusted proof gesture then I don't I
think it's fine there's still of course
this is not something anyone has
successfully implemented at this point
but I think it I just give it as an
example of hope we don't have to do all
the work ourselves right this is exactly
the sort of very boring and tedious task
that is perfect to Outsource to an AI
and this is a way in which less powerful
and less intelligent agents like us can
actually
continue to control and Trust more
powerful ones so build AGI systems that
help us defend against other AGI systems
well for starters
begin with a simple problem of just
making sure that the system that you own
or that's supposed to be loyal to you
has to prove to itself that it's always
going to do the things that you actually
wanted to do right and if it can't prove
it maybe it's still going to do it but
you won't run it so you just forfeit
some aspects of all the cool things AI
can do I I bet you dollars of donuts you
can still do some incredibly cool stuff
for you yeah there are other things too
that we shouldn't sweep under the rug
like not every human agrees on exactly
where what direction we should go with
Humanity right yes and you've talked a
lot about geopolitical things on this on
on your podcast to this effect you know
but
I think that shouldn't distract us from
the fact that there are actually a lot
of things that everybody in the world
virtually agrees on that hey you know
like having no humans on the planet
in a in the near future let's not do
that right you looked at something like
the United Nations sustainable
development goals some of them are
required
ambitious and uh basically all the
countries agree U.S China Russia Ukraine
they all agree so instead of quibbling
about the little things we don't agree
on let's start with the things we do
with Beyond and and get them done
instead of being so distracted by all
these things we disagree on
that warlock wins because
frankly
molok going wild now it feels like a war
on Life playing out in front of our eyes
if you if you just look at it
from space you know we're on this planet
beautiful vibrant ecosystem now we start
chopping down
big parts of it even though nobody most
people thought that was a bad idea
always start doing ocean acidification
wiping out all sorts of species oh now
we have all these close calls we almost
had a nuclear war and we're replacing
more and more of the biosphere with
non-living things we're also replacing
in our social lives a lot of the things
which were so valuable to humanity a lot
of social interactions now are replaced
by people staring into their rectangles
right and I
I'm not a psychologist I'm out of my
depth here but I suspect that part of
the reason why teen suicide and suicide
in general in the US is at
record-breaking levels is actually
caused by
again and so AI Technologies and social
media making people spend less time with
with actual and actually just human
interaction we've all seen
a bunch of good looking people in
restaurants
into the rectangles instead of looking
at each other's eyes right
so that's also a part of the war in life
that that we're we're replacing so many
really life-affirming things
by technology we're we're putting
technology between us
the the technology that was supposed to
connect us is actually distancing us
ourselves from each other and um
and then we're giving ever more power to
things which are not alive these large
corporations are not living things right
they're just maximizing profit
there
I want to win them more in life I I
think we humans together with all our
fellow living things on this planet will
be better off if we can
remain in control over the non-living
things and make sure that they they work
for us I really think it can be done
can you just Linger on this uh maybe
high level philosophical disagreement
with Eliezer yadkowski
bye in this the hope you're stating so
he is very sure
he puts a very high probability
very close to one depending on the day
he puts it at one
that AI is going to kill humans
um
that there's just he does not see a
trajectory which it doesn't end up with
that conclusion what uh which trajectory
do you see that doesn't end up there and
maybe can you
can you see the point he's making
and and can you also see a way out
mm-hmm
first of all I tremendously respect
Elias yukowski and his his thinking
second I do share his view that there's
a pretty large chance that we're not
going to make it as humans there won't
be any humans on the planet
and then not the distance future and
that makes me very sad you know we just
had a little baby and I keep asking
myself you know is um
um
how old is he even gonna get you know
and and um I ask myself hey
it feels I said to my wife recently it
feels a little bit like I was just
diagnosed with some sort of um
cancer which has some you know risk of
of dying from in some risk of surviving
you know
uh
except this is the kind of cancer which
will kill all of humanity so I
completely take seriously his
um his concerns
I think um
but I don't absolutely don't think it's
hopeless I think um there is a there is
a
first of all a lot of momentum now
for the first time actually since the
many many years that have passed since
since I and many others started warning
about this I feel
most people are getting it now
I
I I uh
it's just talking
to this guy in the gas station they were
a house the other day my
and he's like
I think we're getting replaced
and then I think in it so that's
positive that they're finally we're
finally seeing this reaction which is
the first step towards solving the
problem uh second uh I really think that
this this vision of only running ai's
really if the stakes are really high
they can prove to us that they're safe
it's really just virus checking in
Reverse again I I think it's
scientifically doable
I don't think it's hopeless um
we might have to Forfeit some of the
technology that we could get if we were
putting Blind Faith in our AIS but we're
still going to get amazing stuff do you
envision a process with the proof
Checker like something like gpt4 GPT 5
will go through a process no rigorous no
no I think it's hopeless that's like
trying to prove there about five
spaghetti
okay what I think well how the the whole
the vision I have for success is instead
that you know just like we human beings
were able to look at our brains and and
they still out the key knowledge Galileo
when his dad threw him an apple when he
was a kid he was able to catch it
because his brain could in this funny
spaghetti kind of way you know predict
how parabolas are going to move his
Kahneman system one right but then he
got older and it's like wait
this is a parabola it's it's y equals x
squared I can distill this knowledge out
and today you can easily program it into
a computer and it can simulate not just
that but how to get to Mars and so on
right I Envision a similar process where
we use the the amazing learning power of
neural networks
to discover the knowledge in the first
place but we don't stop with a black box
and and use that we then do a second
round of AI where we use automated
systems to extract out the knowledge and
see what is it what are the insights
it's had okay and it's
and then we we put that knowledge into a
completely different kind of um
architecture programming language or
whatever that's that's made in a way
that it can be both really efficient and
also
is more amenable to very formal
verification
that's that's my vision I'm not saying
sitting here saying I'm confident 100
sure that it's going to work you know
but I don't think it's a chance it's
certainly not zero either and it will
certainly be possible to do for a lot of
really cool AI applications that we're
not using now so we can have a lot of
the fun that we're excited about
if we if we do this we're going to need
a little bit of time
that's why it's good to pause and put in
place requirements
one more thing also I I think
you know someone might think well zero
percent chance we're gonna survive let's
just give up right that's very dangerous
because there's no more guaranteed way
it's a fail than to convince yourself
that it's impossible and not to try
you know any if you you know when you
study history in military history the
first thing you learn is that that's
how you do psychological warfare you
persuade the other side that it's
hopeless so they don't even fight
and then then of course you win right
let's not do this uh psychological
warfare on ourselves and say there's a
hundred percent probability we're all
gonna we're all screwed anyway
it's sadly I I do get that a little bit
sometimes from from uh some young people
who are like so convinced that we're all
screwed that they're like I'm just gonna
play game play computer games and do
drugs and because we're screwed anyway
right
it's important
keep the hope alive because it actually
has a causal impact and making it makes
it more likely that we're going to
succeed it seems like the people that
actually build solutions to the problem
seemingly impossible to solve problems
are the ones that believe yeah they're
the ones who are the optimists yeah and
it's like uh it seems like there's some
fundamental law to the universe where
fake it till you make it kind of works
like believe it's possible and it
becomes possible yeah was it Henry Ford
who said that
if you can if you tell yourself that
it's impossible it is
so let's not make that mistake yeah and
this is a big mistake Society is making
or I think all in all everybody's so
gloomy and the media are also very
biased towards if it bleeds it bleeds
and Bloom and doom right so
um
most
visions of the future we have or or a
dystopian which really demotivates
people we want a really really focus on
the upside also to give people the
willingness to fight for it
and um
for AI you and I mostly talked about
Gloom here again but let's not remember
not forget that you know
we have probably both lost
someone we really cared about some
disease that we were told was incurable
well it's not there's no law in physics
saying they have to die of that cancer
or whatever of course you can cure it
and there's so many other things where
that we with a human intelligence have
also failed to solve on this planet
which AI could also very much help us
with right so if we can get this right
just be a little more chill and slow
down a little bit so we get it right
it's mind-blowing how awesome our future
can be right we talked a lot about stuff
on Earth it can be great
but even if you really get ambitious and
look up at the skies right there's no
reason we have to be stuck on this
planet for the rest of um
the remain for billions of years to come
we totally understand now that laws of
physics let life spread out into space
to other solar systems to other galaxies
and flourish for billions of billions of
years
and this to me is a very very hopeful
vision
that really motivates me to to fight
them coming back to in the end something
you talked about again you know this the
struggle how the human struggle is one
of the things that also really gives
meaning to our lives if there's ever
been an epic struggle
this is it and isn't it even more epic
if you're the underdog
if most people are telling you this is
gonna fail it's impossible right and you
persist
and you succeed
right and that's what we can do together
as a species on this one a lot of
pundits are ready to count this out
both in the battle to keep AI safe and
becoming a multiple planetary species
yeah and they're they're the same
challenge if we can keep AI safe that's
how we're gonna get multi-planatory very
efficiently
I had some sort of technical questions
about how to get it right so one idea
uh that I'm not even sure what the right
answer is to is uh should systems like
gpt4 be open sourced in the whole arm
part can you make the can you see the
case for either
I think the answer right now is no
I think the answer early on was yes
so we could bring in the all the
wonderful create the thought process of
everybody on this but asking should we
open source gpt4 now is just the same as
if you say well is it good should we
open source
um
you know how to build really small
nuclear weapons should we open source
how to make bio weapons
to the open source how to make um
a new virus that kills 90 of everybody
who gets it of course we shouldn't
so it's already that powerful
it's already that powerful that we have
to respect the power of the systems
we've built
the knowledge that you get
um
from open sourcing everything we do now
might very well be powerful enough that
people looking at that
can use it to build the things that are
really threatening again let's get it
remember open AI is
gpt4 is a baby AI maybe a sort of baby
Proto almost a little bit AGI according
to what Microsoft's recent paper said
right
it's not that they were scared of what
we're scared about is people taking that
who are
who might be a lot less responsible than
the company that made it right and uh
just going to town with it that's why
we want to it's it's an information
Hazard there are many things which um
you know are not open sourced right now
in society for very good reason
like how how do you make
certain kind of very powerful toxins out
of stuff you can buy at Home Depot you
know you know you don't open source
those things for a reason
and this is really no different
so uh what I'm saying that I have to say
it feels in a bit weird but in a way a
bit weird to say it because MIT is like
the Cradle of the open source movement
and I love open source in general power
to the people let's say um
but um there's always gonna be some
stuff that you don't open source and you
know it's just like you don't open
source so we have a three-month-old baby
right when he gets a little bit older
we're not going to open source to him
all the most dangerous things he could
do in the house yeah right
but it does it's a weird feeling because
this is one of the first moments in
history where so there's a strong case
to be made not to open source software
this is when the software has become
yeah too dangerous yeah but it's not the
first time that we didn't want to open
source a technology technology yeah
is there something to be said about how
to get the release of such systems right
like GPT 4 and gpt5
so open AI went through a pretty
rigorous effort for several months you
could say it could be longer but
nevertheless it's longer than you would
have expected of trying to test the
system to see like what are the ways
goes wrong to make it very difficult for
people
somewhat difficult for people to ask
things how do I make a bomb for one
dollar
or how do I say I hate a certain group
on Twitter in a way that doesn't get me
blocked from Twitter banned from Twitter
those kinds of questions uh so you
basically use the system to do harm yeah
uh is there something you could say
about ideas you have it's just on
looking having thought about this
problem of AI safety how to release such
system how to test such systems when you
have them inside the company
yeah so
a lot of people say that the two biggest
risks from large language models are
it's spreading this information
harmful information the various types
and second being used for offensive
cyber weapon
design I I think those are not the two
greatest threats they're very serious
threats and it's wonderful that people
are trying to mitigate them
it's a much bigger elephant in the room
is how is this it's going to disrupt our
economy in a huge way obviously and
maybe take away a lot of the most
meaningful jobs
and an even bigger one is the one we
spent so much time talking about here
that that
this
becomes the Bootloader for the more
powerful AI write code connected to the
internet manipulate humans yeah and
before we know it we have something else
which is not at all a large language
model it looks nothing like it but which
is way more intelligent than capable and
has goals
and that's the that's the elephant in
the room and and uh obviously no matter
how hard any of these companies have
tried uh they that's not something
that's easy for them to verify with
large language models and the only way
to be
really lower that risk a lot would be
to not let for example try not to never
let it read any code not train on that
and not put it into an API
and um
not
not give it access to so much
information about how to manipulate
humans
so but that doesn't mean you still can't
make a lot
a ton of money on them you know uh we're
gonna
just watch now this coming year right
Microsoft is rolling out the new uh
office suite where you go into Microsoft
Word and give it a prompt they did
writes the whole text for you and then
you edit it
and then you're like oh give me a
PowerPoint version of this and it makes
it and now take the spreadsheet and blah
and you know all of those things I think
are you can debate the economic impact
of it and whether Society is prepared to
deal with this disruption but those are
not the things which
that's not the elephant of the room that
keeps me awake at night for wiping out
Humanity
and
I think that's the biggest
misunderstanding we have a lot of people
think that we're scared of like
automatic spreadsheets that's not the
case that's not what Eleazar was freaked
out about either
is there
in terms of the actual mechanism of how
AI might kill all humans
so something you've been outspoken about
you've talked about a lot is it
autonomous weapon systems
so the use of AI in war is that one of
the things that still you carry concern
for as these systems become more and
more powerful and Gary concern for it
not that all humans are going to get
killed by slaughterbots but rather just
this
express route into orwellian dystopia
where it becomes much easier for very
few to kill very many and therefore it
becomes very easy for very few to
dominate very many right
um
yeah if you want to know how I could
kill all people just ask yourself how we
humans have driven a lot of species
extinct how do we do it
you know we were smarter than them
usually we didn't do it even
systematically by going around
one-on-one one after the other and
stepping on them or shooting them or
anything like that we just like chopped
down their habitat because we needed it
for something else
uh in some cases we did it by putting
more carbon dioxide in the atmosphere
because
of some reason that those animals didn't
even understand and now they're gone
right so if
um if you're in Ai and you just wanna
figure something out then you decide you
know we just really need them the space
here to build more compute facilities
you know
if that's the only goal it has you know
we are just the sort of accidental
roadkill along the way and you could
totally imagine yeah maybe this oxygen
is kind of annoying because it caused
more corrosion so let's get rid of the
oxygen
and good luck surviving after that you
know I I I'm not particularly concerned
that they would want to kill us just
because
that would be like
a goal in itself you know when we
've driven the number we've driven a
number of the elephant species extinct
right it wasn't because we didn't like
elephants
what the basic problem is you just don't
want to give you don't want to seed
control over your planet to some other
more intelligent entity that doesn't
share your goals it's that simple so
which brings us to the other key
challenge which AI Safety Research has
been grappling with for a long time like
how do you make AI
first of all understand our goals and
then adopt our goals and then retain
them as they get smarter right and um
all three of those are really hard right
like
a human child first they're just not
smart enough to understand our goals
we can't even talk
and then eventually they're teenagers
and understand our goals just fine but
they don't share yeah but there is
unfortunately a magic space in the
Middle where they're smart enough to
understand our goals and malleable
enough that we can hopefully with good
parenting and
teach them right from wrong and then
good good goal is still good goals in
them right
so those are all tough challenges with
computers and then you know even if you
teach your kids good goals when they're
little they might outgrow them to and
that's a challenge for machines they
keep improving so these are a lot of
hard hard challenges we're up for but I
don't think any of them are
insurmountable
the fundamental reason why
Eliezer looked so depressed when I last
saw him was because he felt it just
wasn't enough time oh not that it was
unsolvable because there's just not
enough time he was hoping that Humanity
was going to take this threat more
seriously so we would have more time
yeah and now we don't have more time
that's why the open letter is calling
for more time
but even with time the AI alignment
problem
it seems to be really difficult oh yeah
but it's also the most worthy problem
the most important problem for Humanity
to ever solve because if we solve that
one Lex
that aligned AI can help us solve all
the other problems
because it seems like it has to have
constant humility
about his goal constantly question the
goal
because as you optimize towards a
particular goal and you start to achieve
it that's when you have the unintended
consequences all the things you
mentioned about so how do you enforce
and code a constant humility as your
ability become better and better and
better and better Stuart Professor
Stuart Russell Berkeley who is also one
of the
driving forces behind this letter he uh
has a whole research program about this
I think of it as yeah humility exactly
although he calls it inverse
reinforcement learning and other nerdy
terms but it's about exactly that
instead of telling the AI here's this
goal go optimize the bejesus out of it
you tell it okay
do what I want you to do but I'm not
going to tell you right now what it is I
want you to do you need to figure it out
so then you give the incentives to be
very humble and keep asking you
questions along the way is this what you
really meant is this what you wanted and
oh this the other thing I tried didn't
work seemed like it didn't work out
right should I try it differently
what's nice about this is it's not just
philosophical mumbo jumbo it's theorems
and Technical work that with more time I
think it can make a lot of progress and
there are a lot of brilliant people now
working on AI safety and we're just not
we just need to give them a bit more
time but also not that many relative to
the skill of the problem no way exactly
there should be
at least just like every University
worth its name has some cancer research
going on in its biology Department right
every University that's computer that
does computer science should have a real
effort in this area and it's nowhere
near that this is something I hope is
changing now thanks to the gpt4 right so
I I think
if there's a silver lining to um what's
happening here even though I think many
people would wish it would have been
rolled out more carefully is that this
might be the wake-up call
that Humanity needed
to really
stop the stop fantasizing about this
being 100 years off and stop fantasizing
about this being completely controllable
and predictable because
it's so obvious it's it's
not predictable you know why is it that
open that
that I think it was GP chat GPT tried to
persuade a
a journalist what was it before to
divorce his wife you know it was not
because the the engineers had built it
was like
let's put this in here and and screw a
little bit with people
they haven't predicted at all
they built the giant black box and
trained to predict the next word and got
all these emergent properties and oops
it did this you know
um
yeah
I I think this is a very powerful
wake-up call and now anyone watching
this is not scared I would encourage
them to just play a bit more with these
these tools they're out there now like
gpd4
and um
so wake-up call is First Step once
you've broken up uh then gotta slow down
a little bit the risky stuff
to give a chance to all everyone's woken
up to to catch up with us on the safety
front you know what's interesting is you
know MIT
that's computer science but in general
but let's just even say computer science
curriculum
how does the computer science curriculum
change now you mentioned you mentioned
programming yeah like why would you be
when I was coming up programming as a
prestigious position like why would you
be dedicating crazy amounts of time to
become an excellent programmer like the
nature program is fundamentally changing
the nature of our entire education
system
is completely torn on its head has
anyone been able to like load that in
and like think because it's really
turning I mean it's pretty much
professors or English teachers are
beginning to really freak out now yeah
right like to give an essay assignment
then they get back all this fantastic
Pros like this is the style of Hemingway
and then they realize they have to
completely rethink and even you know
just like we stopped teaching um
writing a script is that what you say in
English yeah handwritten yeah yeah when
when everybody started typing you know
like so much of what we teach our kids
today
yeah I mean that's
uh
everything is changing and is exchanging
very it is changing very quickly and so
much of us understanding how to deal
with the big problems of the world is
through the education system and if the
education system is being turned on its
head then what what's next it feels like
having these kinds of conversations is
essential to try to figure it out and
everything is happening so rapidly uh I
don't think there's even speaking of
safety what broad AI safety Define I
don't think most universities have
courses on AI safety it's like
philosophy side and like I I'm an
educator myself so it pains me to see
this say this but I feel our education
right now is the completely obsoleted
by what's happening you know you put a
kid into first grade
and then uh you're envisioning like and
then they're going to come out of high
school 12 years later
and you've already pre-planned now what
they're gonna learn when you're not even
sure if there's going to be any world
left to come out to right
clearly you need to have a much more
opportunistic education system that
keeps adapting itself very rapidly as
Society re-adapts the the skills that
were really useful when the curriculum
was written I mean how many of those
skills are going to get you a job in 12
years I mean seriously if we just Linger
on the gpt4 system a little bit
you kind of hinted at it especially
talking about the importance of
Consciousness in uh in the human mind
with homosexions
do you think gpt4
is conscious
I love this question
so let's define consciousness first
because in my experience like 90 of all
arguments about Consciousness before I
land to the two people arguing having
totally different definitions of what it
is then they're just shouting past each
other
I define consciousness
as subjective experience
right now I'm experiencing colors and
sounds and emotions you know
that does a self-driving car experience
anything
that's the question about whether it's
conscious or not right
other people think you should Define
consciousness differently
fine by me but then maybe use a
different word for it or they can I'm
going to use Consciousness for this at
least
um
so um
but if people hate the yeah so
is gpt4 conscious does GPT 4 have
subjective experience
short answer I don't know
because we still don't know what it is
that gives this wonderful subjective
experience
that is kind of the meaning of our life
right because meaning itself the feeling
of meaning is a subjective experience
Joy is a subjective experience love is a
subjective experience
we don't know what it is
I've written some papers about this a
lot of people have
Julio tonone Professor has
stuck his neck out the farthest and
written down actually very bold
mathematical
conjecture for what's the essence of
conscious information processing he
might be wrong he might be right
we should test it
he postulates the Consciousness has to
do with loops in the information
processing so our brain has loops
information you can go around and around
in in computer science nerdspeak you
call it a recurrent neural network where
some of the output gets fed back in
again
and with his
mathematical formalism if it's a feed
forward neural network where information
only goes in One Direction
like from your eye retina into the back
of your brain for example that's not
conscious so he would predict that your
retina itself isn't conscious of
anything
or a video camera now the interesting
thing about gpt4 is it's also one-way
flow of information so if tanoni is
right and gpt4 is a very intelligent
zombie they can do all this smart stuff
but isn't experiencing anything
and this is
both a relief in that you don't have if
it's true you know in that you don't
have to feel guilty about turning off
gpt4 and wiping its memory whenever a
new user comes along
I wouldn't like if someone you did that
to me neutralized me like in Men In
Black
but it's also
creepy that you can have very high
intelligence perhaps then it's not
conscious because if we get replaced by
machines
and by the sad enough that Humanity
isn't here anymore because I kind of
like Humanity
but at least if the machines were
conscious they could be like well but
they are descendants and maybe we they
have our values they're our children but
if if tenoni is right and it's all these
are all
trans Transformers that are
not in the sense of the of Hollywood but
in the sense of these one-way Direction
and neural networks so they're all the
zombies that's the Ultimate Zombie
Apocalypse Now we have this universe
that goes on with great construction
projects and stuff but there's no one
experiencing anything that would be like
the ultimate depressing future
so I actually think uh
as we move forward with building more
advanced AI
should do more research on figuring out
what kind of information processing
actually has experience because I think
that's what it's all about and I
completely don't buy
the dismissal that some people some
people will say well this is all
because Consciousness equals
intelligence right that's obviously not
true you can have a lot of conscious
experience when you're not really
accomplishing any goals at all you're
just reflecting on something and you can
sometimes um
have things doing things that are quite
intelligence probably without being
being conscious but I also worry that we
humans won't
will discriminate against AI systems
that clearly exhibit Consciousness that
we will not allow AI systems
self-consciousness we'll come up with
theories
about measuring Consciousness that will
say this is a lesser being and this was
like I worry about that because maybe we
humans will create something that is
better than us humans in the in the way
that we find beautiful which is they
they cut they have a deeper subjective
experience of reality not only are they
smarter but they feel deeper and we
humans will hate them for it
as we as human history has shown
they'll be the other will try to
suppress it they'll create conflict
they'll create War all of this I I worry
about this too are you saying that we
humans sometimes come up with
self-serving arguments no we would never
do that would we well that's the danger
here is uh even in this early stages we
might create something beautiful yeah
and uh will erase its memory I I
was horrified as a kid when someone
started boiling uh
boiling lobsters like oh my God that
that's so cruel and some grown-up there
back in Sweden said oh it doesn't feel
pain I'm like how do you know that oh
scientists have shown that
and then there was a recent study where
they showed that lobsters actually do
feel pain
when you boil them so they banned
Lobster boiling in Switzerland now you
have to kill them in a different way
first
so it's presumably scientific research
boil down to someone asked the lobster
does this hurt
survey so we do the same thing with
cruelty to farm animals also all these
self-serving Arguments for why they're
fine and yeah so we should certainly
what the watchful I think step one is
just be humble and acknowledge that
Consciousness is not the same thing as
intelligence
and I believe that Consciousness still
is a form of information processing
where it's really information being
aware of itself in a certain way and
let's study it and give ourselves a
little bit of time and I think we will
be able to figure out actually what it
is
that causes Consciousness and then we
can make probably unconscious robots
that do the boring jobs that we would
feel immoral to give the machines but if
you have a companion robot taking care
of your mom
or something like that she would
probably want it to be conscious right
so the emotions it seems to display
aren't fake
all these things can be
done in a good way if we give ourselves
a little bit of time and don't run and
take on this challenge is there
something you could say to the timeline
that you think about uh about the
development of AGI
depending on the day I'm sure that
changes for you but when do you think
there'll be a really big leap in
intelligence where it would definitively
say we have built AGI do you think it's
one year from now five years from now 10
20 50.
what's Your Gut say
honestly
for the past decade I've deliberately
given very long timelines because I
didn't want to fuel some kind of stupid
Morlock race yeah but I think that cat
has really left the bag now
uh I I think it might be very very close
I I don't think that Microsoft paper is
totally off when they say that there are
some glimmers of AJ it's not AGI yet
it's not an agent there's a lot of
things it can't do but um
I wouldn't bet very strongly against it
happening
very soon that's why we decided to do
this open letter because you know if
there's ever been a time to pause
uh you know it's today
there's a feeling like this gpt4 is a
big transition into waking everybody up
to uh the effectiveness of these systems
yeah and so the next version
will be big yeah and if that next one
isn't AGI maybe the next next one will
and there are many companies trying to
do these things
um the basic architecture of them is not
some sort of super well kept secret so
this is this is the time to um
a lot of people have said for many years
that there will come a time when we want
to pause a little bit
that time is now
you have spoken about and thought about
nuclear war a lot
uh over the past year with
seemingly have come
closest to the precipice of nuclear war
then at least in my lifetime
yeah what do you learn about human
nature from that it's our old friend
moloch again it's really scary to see it
where
America doesn't want there to be a
nuclear war Russia doesn't want to be a
global nuclear war either we know we
both know that it would just be another
if we just try to do it if both sides
try to launch first it's just another
suicide race right so why are we why is
it the way you said that this is the
closest we've come since 1962. in fact I
think we've come closer now than even
the Cuban Missile Crisis it's because of
moloch you know you you have these other
forces
on one hand you have um the West
saying that uh we have to drive Russia
out of Ukraine it's a matter of pride
and we've staked so much on it that it
would be seen as a huge
loss of The credibility of the West if
we don't drive Russia out entirely of
the Ukraine
and on the other hand you have Russia
who um
has and you have the Russian leadership
who knows that if they get completely
driven out of Ukraine you know
it might it's not just going to be very
humiliating
for them but they might it often happens
when countries lose Wars that things
don't go so well for their leadership
either like you remember when Argentina
invaded the Falkland Islands
the the military Junta that ordered that
right people were cheering on the
streets at first when they are when they
took it
and then when they got their butt kicked
by the British you know what happened to
those guys
they were out and I believe those were
still alive or in jail now right so so
you know the Russian leadership is
entirely cornered where they know that
uh just
getting driven out of Ukraine is not an
option
um and um
so this to me is a typical example of
molecu you have these incentives of the
two parties
where both of them are just driven to
escalate more and more right if Russia
starts losing in the conventional
Warfare
the only thing they can do with since
the back against the war is to keep
escalating
and but and the West has put itself in
the in the situation now we're sort of
already committed that the drive rush
out so the only option the West has is
to call Russ's Bluff and keep sending in
more weapons
this really bothers me because moloch
can sometimes Drive competing parties to
do something which is ultimately just
really bad for both of them
and uh you know what makes me even more
worried is not just that I
it's
difficult to see an ending
a quick peaceful ending to this tragedy
that doesn't involve some horrible
escalation
um but also that we understand more
clearly now just how horrible it was
gonna it would be
there was an amazing paper that was
published in nature food this uh August
by some of the top researchers who've
been studying nuclear winter for a long
time and what they basically did was
they combined
climate models
with
food agricultural models so instead of
just saying yeah you know it gets really
cold blah blah they figured out actually
how many people would die in the
different different countries
and it's uh it's pretty mind-blowing you
know so basically what happens you know
is the thing that kills the most people
is not the explosions it's not the
radioactivity it's not the EMP Mayhem
it's not the right
the rampaging mobs forcing food no it's
it's the fact that you get so much smoke
coming up from the burning cities into
the stratosphere that um
that spreads around the earth from the
jet streams so
in typical models you get like 10 years
or so where it's just crazy cold and
during the first year or after the the
war and their models
the temperature drops in in Nebraska and
in the Ukraine bread baskets you know by
like
20 Celsius or so if I remember
no yeah 2030 Celsius depending on where
you are 40 so it's just in some places
which is you know 40 Fahrenheit to 80
Fahrenheit colder than what it would
normally be
so you know I'm not good at farming but
uh it's knowing if it drops below
freezing pretty much almost days in July
and then like that's not good so they
worked out they put this into their
farming models and what they found was
really interesting the countries that
get the most hard hit are the ones in
the northern hemisphere
so in in the U.S
and and one model they had about 99 of
all Americans starving to death
in Russia and China and Europe also
about 99 98 starving to death
so you might be like oh it's kind of
poetic justice that both the Russians
and the Americans
99 of them have to pay for it because it
was their bombs that did it but you know
that doesn't particularly cheer people
up in Sweden or other random countries
that have nothing to do with it right
and um
hit the
I think it hasn't entered the mainstream
uh
not understanding very much just like
how bad this is uh most people
especially a lot of people in decision
making positions still think of nuclear
weapons as something that makes you
powerful
uh scary powerful they don't think of it
as something where uh yeah just
to within a percent or two you know
we're all just gonna starve to death
and um and starving to death is
is uh
the worst way to die as Audemars all all
the famines in history show
the torture involved in that probably
brings out the worst in people also when
when people are desperate like this it's
not so some people I've heard some
people say that
if that's what's going to happen there'd
rather be Ground Zero and just get
vaporized you know
and but uh
so but I think people underestimate the
risk of this because they they
aren't afraid of Morlock they think oh
it's just gonna be because humans don't
want this so it's not going to happen
that's the whole point the mark that
things happened that nobody wanted and
that applies to nuclear weapons and that
applies to AGI
exactly and it applies to some of the
things that people have gotten most
upset with capitalism for also right
where everybody was just kind of Trapped
you know
it's not to see if some company does
something
it causes a lot of harm and not that the
CEO is a bad person but she or he knew
that you know that the other all the
other companies were doing this too so
Morlock is um
as a former foe I hope for someone
making
good movies so we can see who the real
enemy is so we don't because we're not
fighting against each other molok makes
us fight against each other that small
that's what Malik's superpower is
The Hope here is any kind of technology
or other mechanism that lets us instead
realize
that we're finding The Wrong Enemy right
now it's such a fascinating battle it's
not a sport system it's US versus it
yeah yeah we are fighting Malik for
human survival we as a civilization have
you seen the movie Needful Things
it's a Stephen King novel I love Stephen
King and uh Max fonseedov Swedish actors
playing the guys it's brilliant exactly
I just thought I hadn't thought about
that until now but that's the closest
I've seen to a a movie about moloch I
don't want to spoil the film for anyone
who wants to watch it but basically
it's about this guy who turns out that
you can interpret him as the devil or
whatever but he doesn't actually ever go
around and kill people or torture people
with burning coal or anything
he makes everybody fight each other it
makes everybody hate fear each other
hate each other and then kill each other
so that that's the movie about malloc
you know love is the answer that seems
to be
um
one of the ways to fight
Malik is by
um compassion by seeing the common
Humanity yes yes and to not sound so we
don't sound like like uh what's the
Kumbaya Tree Huggers here right
we're not just saying love and peace man
we're we're trying to actually help
people understand the true facts
about the other side
and feel the compassion
because
the truth makes you more compassionate
right
so I I that's why I really like using AI
for truth and for truth seeking
Technologies can
that can as a result you know get us
more love than hate and
and even if you can't get love you know
settle for settle for some understanding
which already gives compassion if
someone is like you know
I really disagree with you
Lex but I can see why you're where
you're coming from you're not a
bad person who needs to be destroyed but
I disagree with you and I'm happy to
have an argument about it you know
that's a lot of progress compared to
where we are 2023 in the public space
wouldn't you say
if we solve the AI safety problem as
we've talked about and then you Max tag
Mark who has been talking about this
uh for many years get to sit down with
the AGI with the early AGI system on a
beach with a drink
what kind of what would you ask her what
kind of question would you ask what
would you talk about
something so much smarter than you
a really Zinger of a question that's a
good one would you be afraid
to ask some questions no
see I'm not afraid of the truth I'm very
humble I know I'm just a meat pie you're
with all these flaws you know but yeah I
I have that we talked a lot about Homo
sentience I really already tried that
for a long time with myself just so that
is what's really valuable about being
alive for me is that I have these
meaningful experiences
it's not that I'm
have what I'm good at this or good at
that or whatever because there's so much
I suck at then
so you're not afraid for the system to
show you just how dumb you are no no in
fact my son reminds me of that
you could find out how dumb you are in
terms of physics how little how little
we humans understand I'm cool with that
I think I think um
so I I can't waffle my way out of this
question it's a fair one it was tough
I think given that I'm a really really
curious person that's
really the defining part of who I am I'm
so curious
uh I have some Physics questions
I loved I love to understand I have some
questions about Consciousness about the
nature of reality or just really really
love to understand also
I can tell you one for example that I've
been obsessing about a lot recently
so I believe that
it's supposed to Noni is right I suppose
there are some information processing
systems that are conscious and some are
not suppose you can even make reasonably
smart things like gpt4 that are not
conscious but you can also make them
conscious
here's the question that keeps me make
it might
is it the case that the unconscious
zombie systems that are really
intelligent are also really efficient so
they're really inefficient
so that when you try to make things more
efficient we still naturally be a
pressure to do they become conscious
I'm kind of hoping that that's correct
and I do you want me to give you a hand
wavy argument for it you know like
in my lab again every time we look at
how it how these large language models
do something we see that they do them in
really dumb ways and you could make it
make it better if if you uh
we have loops in our computer language
for a reason
the code which would get way way longer
if you weren't allowed to use them it's
more efficient to have the loops
and
in order to have self-reflection whether
it's conscious or not right even an
operating system knows things about
itself right
you need to have loops already and so I
think this is I'm waving my hands a lot
but I suspect that um
the most efficient way of implementing a
given level of intelligence
has Loops in it the self-reflection it
can
and will be conscious isn't that great
news yes if it's true it's wonderful
because then we don't have to fear the
ultimate zombie apocalypse and I think
if you look at our brains actually our
brains are part zombie
and part conscious
when I open my eyes
I immediately
take all these pixels and hit my ref on
my retina right and like oh that's Lex
but I have no freaking clue of how I did
that computation it's actually quite
complicated right it was only relatively
recently we could even do it well with
machines right you get a bunch of
information processing happening in my
retina and then it goes to the lateral
nucleus my Thalamus into the area V1 V2
V4 and the fusiform face area here that
Nancy can wish her MIT invented and blah
blah blah
I'd have no freaking clue how that
worked right it feels to me subjectively
like my conscious module
just got a little email it's like
face facial processing fit
task complete it's Lex yeah I'm gonna
just go with that right so this fits
perfectly with tanoni's model because
this was all one way with the
information processing mainly
and uh it turned out for that particular
task that's all you needed and it
probably was kind of the most efficient
way to do it but there are a lot of
other things that we associate with
higher intelligence and planning and and
so on and so forth where you kind of
want to have loops and be able to
ruminate and self-reflect and introspect
and so on where
my hunch is that if you want to fake
that with a zombie system that just all
goes one way you have to like unroll
those loops and it just really really
long and it's much more inefficient
so I'm actually hopeful that AI if in
the future we have all these various
Sublime and interesting machines that do
cool things
and are aligned with that
they will be at least they will also
have Consciousness for the kind of these
things that we do
the great intelligence is also
correlated to Great Consciousness or a
deep kind of Consciousness yes so that's
a happy thought for me because the
zombie of a couple of apocalypse really
is my worst nightmare of all it would be
like adding insult to injury not only
did we get replaced but we freaking
replaced ourselves by zombies like how
dumb can we be that's such a beautiful
vision and that's actually a provable
one that's one that we humans can Intuit
and prove that those two things are
correlated as we start to understand
what it means to be intelligent and what
it means to be conscious which these
systems uh early agi-like systems will
help us understand and I just want to
say one more thing this is super
important most of my colleagues when I
started going on in my Consciousness
tell me that it's all and I
should stop talking about it
I hear a little inner voice from my
father and for my mom saying keep
talking about it because I think they're
wrong and and and
the main
way to convince people like that that
they're wrong if they say that
Consciousness is just equal to
intelligence just ask them what's wrong
with torture or why are You Against
torture
if it's just about you know
these these particles are moving this
way rather than that way
and there is no such thing as subjective
experience what's wrong with torture I
mean do you have a good comeback to that
no it seems like suffering suffering
imposed onto other humans is somehow
deeply wrong in a way that intelligence
doesn't quite explain that if someone
tells me well you know
It's Just an Illusion Consciousness
whatever you know
I would like to invite them the next
time they're having surgery to do it
without anesthesia
what is anesthesia really doing if you
have it you can have it local anesthesia
when you're awake I have that when they
fix my shoulder right it's super
entertaining uh
what was that that it did it just
removed my subjective experience of pain
it didn't change anything it was
actually happening in my shoulder right
so if someone says that's all
and skip the anesthesia as my advice
this is incredibly Central
it could be fundamental to whatever this
thing we have going on here it is
fundamental because we're we what we
feel is so fundamental is suffering and
joy and pleasure and meaning and
that's all those are all subjective
experiences there
and let's not those are the elephant in
the room that's what makes life worth
living and that's what can make it
horrible if it's just a bunch of
suffering so let's not make a mistake of
saying that that's all
and let's not make the mistake of uh
not instilling the AI systems with that
same
thing that makes us uh special yeah
Max uh it's a huge honor that you will
sit down to me the first time
uh on the first episode of this podcast
it's a huge honest sit down with me
again and talk about this what I think
is uh the most important topic
the most important problem that we
humans have to face and hopefully solve
yeah well the honor is all mine and I'm
so grateful to Youth for making more
people aware of the fact that Humanity
has reached the most important fork in
the road ever and it's history and
that's
turn in the correct direction
thanks for listening to this
conversation with Max tagmark to support
this podcast please check out our
sponsors in the description and now let
me leave you some words from Frank
Herbert
history is a constant race between
invention and catastrophe
thank you for listening and hope to see
you next time