Kind: captions
Language: en
the following is a conversation with
Ariane Vinnie Alice he's a senior
research scientist at google deepmind
and before that he was a Google brain
and Berkeley his research has been cited
over 39,000 times he's truly one of the
most brilliant and impactful minds in
the field of deep learning
he's behind some of the biggest papers
and ideas and AI including sequence the
sequence learning audio generation image
captioning neural machine translation
and of course reinforcement learning
he's a lead researcher of the Alpha Star
project creating an agent that defeated
a top professional at the game of
StarCraft this conversation is part of
the artificial intelligence podcast if
you enjoy it subscribe on youtube itunes
or simply connect with me on twitter at
Lex Friedman spelled Fri D and now
here's my conversation with Arielle
Minnie Alice
you spearheaded the deepmind team behind
alpha star that recently beat a top
professional player Starcraft so you
have an incredible wealth of work and
deep learning in a bunch of fields but
let's talk about Starcraft first let's
go back to the very beginning even
before alpha star before deep mine
before deep learning first what came
first for you a lot for programming or a
love for videogames I think for me it
definitely came first the drive to play
videogames I really liked computers I
didn't really code much but what I would
do is I would just mess with the
computer break it and fix it that was
the level of skills I guess that I
gained in my very early days I mean when
I was 10 or 11 and then I really got
into video games especially Starcraft
actually the first version I spent most
of my time just playing kind of pseudo
professionally as professionally as you
could play back in 98 in Europe which
was not a very main scene like that
what's called nowadays eSports right of
course in the 90s so how did you get
into StarCraft what was your favorite
race how do you develop how did you
develop your skill what was your
strategy all that kind of thing so as a
player I tended to try to play not many
games not to kind of disclose the
strategies that I kind of developed and
I like to play random actually not in
competitions but just to I think in
StarCraft there's well there's three
main races and I found it very useful to
play with all of them and so I would
choose random many times even sometimes
in tournaments to gain skill on the
three races because it's not how you
play against someone but also if you
understand the race because you play it
you also understand what's annoying what
then when you're on the other side what
to do to annoy that person to try to
gain advantages here and there and so on
so I actually played random although I
must say in terms of favorite race I
really liked zerk I was probably best at
Zerg and that's probably what I tend to
use towards the end of my career
year before starting University so let's
step back a little bit could you try to
describe Starcraft to people that may
never have played video games especially
the massively online variety right so
craft so Starcraft is a real-time
strategy game and the way to think about
Starcraft perhaps if you understand a
bit chess is that there are there's a
board which is called map or or or the
gallic the map where people play against
each other there's obviously many ways
you can play but the most interesting
one is the one versus one setup where
you just play against someone else or
even the built in AI right the wizard
put a system that can play the game
reasonably well if you don't know how to
play and then in this board you have
again pieces like in chess but these
pieces are not there initially like they
are in chess you actually need to decide
to gather resources to decide which
pieces to build so in a way you're
starting almost with no pieces you start
gathering resources in StarCraft there's
minerals and gas that you you can gather
and then you must decide how much do you
want to focus for instance on gathering
more resources or starting to build
units or pieces and then once you have
enough pieces or maybe like attack you
know a good attack composition then you
go and attack the other side of the map
and now the other main difference with
chess is that you don't see the other
side of the map so you're not seeing the
moves of the enemy it's what we call
partially observable
so as a result you must not only decide
Trading of economy versus building your
own units but you also must decide
whether you want to scout
to gather information but also by
scouting you might be giving away some
information that you might be hiding
from the enemy so there's a lot of
complex decision-making all in real-time
there's also unlike chess this is not a
turn-based game you play basically all
the time continuously and thus some
skill in terms of speed and accuracy of
clicking is also very important and
people that train for these really play
this game at an amazing skill
I've seen many times these and if you
can witness this life is really really
impressive so in a way it's kind of a
chess where you don't see the other side
of the board you're building your own
pieces and you also need to gather
resources to basically get some money to
build other buildings pieces technology
and so on
from the perspective of the human player
the difference between that and chess or
maybe that and a game like turn-based
strategy like heroes a might of magic is
that there's an anxiety because you have
to make these decisions really quickly
and if you are not actually aware or
what decisions work it's a very
stressful balance the if there's
everything you describe is actually
quite stressful difficult to balance for
a mature human player I don't know if it
gets easier at the professional level
like if they're fully aware what they
have to do but at the amateur level
there's this anxiety oh crap I'm being
attacked oh crap I have to build up
resource oh I have to probably expand
and all these the time the real-time
strategy aspect is really stressful and
computation I'm sure difficult we'll get
into it but for me battlenet so
Starcraft was released in 98 20 years
ago which is hard to believe and
Blizzard battlenet with Diablo 96 came
out and to me it might be a narrow
perspective but it changed online gaming
and perhaps society forever
yeah but I may have made way too narrow
viewpoint but from your perspective can
you talk about the history of gaming
over the past 20 years is this how
transformational how important is this
line of games right so I think I I kind
of was an active gamer whilst this was
developing the internet I'm online
gaming so for me that the way it came
was I played other games strategy
related I play a bit of common and
conquer and then I played Warcraft 2
which is from Blizzard but at the time I
didn't know I didn't understand about
what Blizzard was or anything Warcraft 2
was just a game
which is which was actually very similar
to start off in many ways it's also
real-time strategy game and where
there's orcs and humans so there's only
two races throughs offline and it was
offline right so I remember a friend of
mine came to to school say oh there's
this new cool game called Starcraft and
I just said all these sounds like just a
copy of Warcraft 2 until I kind of
installed it and at the time I am from
Spain so we didn't have internet like
very good internet right so there was
for us a Starcraft became first kind of
an offline experience where you kind of
start to play these missions right you
play against some sort of scripted
things to develop the story of the
characters in the game and then later on
I start playing against the butene AI
and I thought it was impossible to
defeat it then eventually you defeat one
and you can actually break n7 built in
the eyes at the same time which also
felt impossible but actually it's not
that hard to beat seven built-in eyes at
once so once we achieve that also we
discovered that we could play as I said
internet wasn't that great but we could
play with the land right on like
basically against each other if we were
in the same place because you could just
connect machines with like cables right
so we started playing in LAN mode and
again you know as a group of friends and
it was really really like much more
entertaining than playing against the
eyes and later on as internet was
starting to develop and being a bit
faster and more reliable then it's when
I started experiencing battlenet which
is these amazing universe not only
because of the fact that you can play
the game against any way anyone in the
world but you can also get to know more
people
you just get exposed to now like this
vast variety of it's kind of a bit when
the chats came about right there was a
chat system you could play against
people where you could also chat with
people not only about soccer but about
anything and that became a way of life
for kind of two years and obviously then
it became like an elite exploded didn't
mean that I started to play more
seriously going to tournaments and so on
so forth
do you have a sense and a societal
sociological level what
whole part of society that many of us
are not aware of and it's a huge part of
society which is gamers I mean every
time I come across that in YouTube or
streaming sites I mean this is a huge
number of people play games religiously
do you have a sense of those folks
especially now that you've returned to
that realm a little bit on the high side
yeah so in fact I even after soccer if I
actually played World of Warcraft which
is mainly the main sort of online world
or in presence that you get to interact
with lots of people so I played that for
a little bit it was to me it was a bit
less stressful than Starcraft because
winning was kind of a given you just put
in this world and you can always
complete missions but I think it was
actually the social aspect of especially
Starcraft first and then games like
World of Warcraft really shaped me in a
very interesting ways because you had
you get to experience it's just people
you wouldn't usually interact with right
so even nowadays I still have many
Facebook friends from the area where I
played online and their ways of thinking
is even political they just don't we
don't live in like we don't interact in
there in the real world but we were
connected by basically fiber and that
way I actually get to understand a bit
better that we live in a diverse world
and these were just connections that
were made by because you know I happened
to go in a city in a virtual city as a
priest and I met these you know this
warrior and we became friends and then
we start like playing together right so
I think it's it's it's transformative
and more and more and more people are
more aware of it I mean it's it's
becoming quite mainstream but back in
the day as you were saying in 2000 2005
even it was very still very strange
thing to do especially in in Europe I
think there were exceptions like Korea
for instance it was amazing like that
that everything happened so early in
terms of cyber cafes like it's if you go
to Seoul it's a city that back in the
day Starcraft was kind of you could be a
celebrity by playing Starcraft but this
was like 99 2000 right it's not like
recently
so um yeah it's quite it's quite
interesting to to look back and and yeah
I think it's changing society that the
same way of course like technology and
social networks and so on are also
transforming things and a quick tangent
let me ask you're also one of the most
productive people in your particular
chosen passion and path in life and yet
you're also appreciate and enjoy video
games do you think it's possible to do
to enjoy video games in moderation
someone told me that you could choose
two out of three when I was playing
video games you could choose having a
girlfriend playing video games or
studying and I think for the most part
it was relatively true these things do
take time games like stark if you take
the game pretty seriously and you wanna
study it then you obviously will
dedicate more time to it and I
definitely took gaming and obviously
studying very seriously I loved learning
science and etc so to me especially when
I started University undergrad I kind of
step off Starcraft I actually fully
stopped playing and then wall of war
curve was a bit more casual you could
just connect online and I mean it was it
was fun but I as I said that was not as
much time investment as it was for me in
StarCraft ok so let's get into alpha
star what are the your behind the team
so deep mine has been working on
Starcraft and released a bunch of cool
open-source agents and so on the past
few years but alpha star really is the
moment where the first time you beat a
world-class player so what are the
parameters of the challenge in the way
that alpha star took it on and how did
you and David and the rest of deepmind
team get into it
consider that you can even beat the best
in the world or top players I think it
all started in back in 2015
actually I'm lying I think it was 2014
when the mine was acquired by Google and
I at the time was at Google brain which
is it it was in California
in California we had this summit where
we got together the two groups so Google
brain and google deepmind got together
and we gave a series of talks and given
that they were doing deep reinforcement
learning for games I decided to bring up
part of my past which I had developed at
Berkeley like this thing which we called
Berkeley over mine which is really just
a Starcraft one but right so I about
that and I remember them is just came to
me and said well maybe not now it's it's
perhaps a bit too early but you should
just come to the mine and do this again
with deep reinforcement learning right
and at the time it sounded very
science-fiction for for several reasons
but then in 2016 when I actually moved
to London and joined in mind
transferring from brain it became
apparent that because of the alphago
moment and kind of Blizzard reaching out
to us to say wait like do you want the
next challenge and also me being
full-time at deep mine it's a sort of
kind of all these came together and then
I was I went to to air vine in
California to the Blizzard headquarters
to just chat with them and try to
explain how would it all work before you
do anything and the approach has always
been about the learning perspective
right so in in Berkeley we did a lot of
rule-based you know conditioning and or
if you have more than three units then
go attack and if the other has more
units than me I retreat and so on and so
forth and of course the point of deep
reinforcement learning deep learning
machine learning in general is that all
these should be learned behavior so that
kind of was the DNA of the project since
its inception in 2016 where we just
didn't even have an environment to work
with and so these that's how it all
started really so if you go back to a
conversation with damage or even in your
own head how far away did you because
that's we're talking about Atari games
we're talking about go which is kind of
if you're honest about it really far
away from Starcraft in a well now that
you've beaten it maybe you could say
it's close but is it's much it seems
like Starcraft is way harder than go
philosophically in mathematically
speaking so how far away did you did you
think you were do you think it's 2019 in
18 you could be doing as well as you
have yeah when I when I kind of thought
about okay I'm gonna dedicate know a lot
of my time and focus on this and
obviously I do a lot of different
research in deep learning so spending
time on it I mean I really had to kind
of think there's gonna be something good
happening out of this so really I
thought well this sounds impossible and
it probably is impossible to do the full
thing like the all like the full game
where you play one versus one and it's
only a neural network playing and so on
so it really felt like I just didn't
even think it was possible but on the
other hand I could see some stepping
stones like towards that goal clearly
you could define subproblems in
StarCraft and sort of dissect it a bit
and say okay here is a part of the game
here's another part and also obviously
the fact so this was really also
critical to me the fact that we could
access human replays right so Blizzard
was very kind and in fact they
open-source these for the whole
community where you can just go and it's
not every single Starcraft game ever
played but it's a lot of them you can
just go and download and every day they
will you can just query a dataset and
say well give me all the games that were
played today and given my kind of
experience with language and sequences
and supervised learning I thought well
that's definitely gonna be very helpful
and something quite unique now because
ever before we had such a large dataset
of replays of people playing the game at
this scale of such a complex video game
right so that to me was a precious
resource and as soon as I knew that
Blizzard was able to kind of give these
to the community I started to feel
positive about something non-trivial
happening but but I also thought the
full thing like really no rules no no
single line of code that tries to say
well I mean if you see this unit will
the detector all these not having any of
these specializations seemed really
really really difficult to me
I do also like that Blizzard was teasing
or even trolling you sort of almost yeah
pulling you in into this really
difficult challenge they have any aware
and that what's what's the interest from
the perspective of Blizzard except just
curiosity yeah I think Blizzard has
really understood and and really bring
bring forward these competitiveness of
eSports in games the Starcraft really
kind of sparked a lot of like something
that almost was never seen especially as
I was saying he back in Korea so they
just probably thought well this is such
a pure 1vs1 setup that it would be great
to see if something that can play Atari
or go and then later on chess could
could even tackle these kind of complex
realtime strategy game right so for them
they wanted to see first obviously
whether it was possible if the game they
created was in a way solvable to some
extent and I think on the other hand
they also are a pretty modern company
that innovates a lot so just starting to
understand AI for them to how to bring
AI into games is not is not AI for games
but get a games for AI oh right I mean
both ways I think can work and you up we
obviously did mine use games for AI
right to drive AI progress but Blizzard
might actually be able to do and many
other companies to start to understand
done to the opposite so I think that is
also something they can get out of these
and they definitely we have brainstorm a
lot about about these right but one of
the interesting things to me about
Starcraft and Diablo and these games
that blizzards created is the task of
balancing classes for example sort of
making the game fair from the starting
point and then let's skill the term and
the outcome is there uh I mean can you
first comment there's three races Zerg
Protoss and Terran I don't know if I've
ever said that out loud is that how you
pronounce it Terran yeah yeah I don't
think I've ever seen personally interact
with anybody about Starcraft it's funny
so they seem to be pretty balanced
I wonder if the AI the work that you're
doing with the Alpha star would help
balance them even further is that
something you think about is that
something that Blizzard is thinking
about right so so balancing when you add
a new unit or a new spell type is
obviously possible given that you can
always train or retrain at scale some
agent that might start using that in
unintended ways but I think actually if
you understand how StarCraft has kind of
co-evolved with players in a way I think
it's actually very cool the ways that
many of the things and strategies that
people came up with right so I think
it's we've seen it over and over in
StarCraft that Blizzard comes up with
maybe a new unit and then some players
get creative and do something kind of
unintentional or something that Blizzard
designers that just simply didn't test
or think about and then after that
becomes kind of mainstream in the
community Blizzard patches the game and
and then they kind of maybe weaken that
strategy or or make it actually more
interesting but a bit more balanced so
these kind of continual talk between
players and Blizzard is is kind of what
has defined them actually in actually
most games that the in in stykera but
also in World of Warcraft they would do
that there are several classes and it
would be not good that everyone plays
absolutely the same race or and so on
right so I think they they do care about
balancing of course and they do a fair
amount of testing but it's also
beautiful to to also see how players get
creative anyways and I mean whether I
can be more creative at this point I
don't think so right I mean it's just
sometimes something so amazing happens
like I remember back in in the days like
you have these drop ships that could
drop rivers and that was actually not
thought about that you could drop this
unit that has this what's called Splash
Damage that would basically eliminate
all the enemies workers at once no one
thought that you could actually put them
and really early game do that kind of
damage and then we know things change in
the game but I don't know I think
there's it's quite an amazing
exploration process from both sides
players and Blizzard alike well
it's it's almost like a reinforcement
learning exploration but a the scale of
humans that play that play Blizzard
games is almost on the scale of a
large-scale deepmind RL experiment I
mean if you look at the numbers that's I
mean you're talking about I don't know
how many games but hundreds of thousands
of games probably a month yeah I mean so
that you could it's almost the same as
running RL agents what an aspect of the
problem of StarCraft II things the
hardest is it the like you said the
imperfect information is it the fact
they have to do long term planning is it
the real time aspects we have to do
stuff really quickly is it the fact that
a large action space that you can do so
many possible things or is it you know
in the game theoretic sense there is no
Nash equilibria at least you don't know
what the optimal strategy is because
there's way too many options right
what's is there something that stands
out is just like the hardest the most
annoying thing so when we sort of looked
at the problem and start to define
permit like the parameters of it right
what are the observations what are the
actions it became very apparent that you
know that the very first barrier that
one would hit in StarCraft would be
because of the action space being so
large and as not being able to search
like you could in in chess or or go even
though the search space is vast the main
problem that we identified was that of
exploration right so without any sort of
human knowledge or human prior if you
think about StarCraft and you know how
deep reinforcement learning is algorithm
works work which is essentially by
issuing random actions and hoping that
they will get some wins sometimes so
they could learn so if you think of the
of the action space in StarCraft almost
anything you can do in the early game is
bad because any action involves taking
workers which are mining minerals for
free that's something that the game does
automatically sends them to mine and you
would immediately just take them out of
mining and send them around so just
thinking how how is it gonna be possible
to to get to understand the
these concepts but but even more like
expanding right there's there's these
buildings you can place in other
locations in the map to gather more
resources but the location of the
building is important and you have to
select a worker send it walking to that
location build the building wait for the
building to be built and then put extra
workers there so they start mining that
just that feels like impossible if you
just randomly click to produce that
state desirable state that then you
could hope to learn from because
eventually that may yield to an extra
win right so for me the exploration
problem and due to the actions pace and
the fact that there's not really turns
there's so many turns because the game
essentially peaks at 22 times per second
if you mean that's how they keep
discretize sort of time obviously you
always have to discretize time there's
not no such thing as real time but it's
really a lot of time steps of things
that could go wrong and that definitely
felt a priori like the hardest you
mentioned many good ones I think partial
observability and the fact that there is
no perfect strategy because of the
partial observability those are very
interesting problems we start seeing
more and more now in terms of us we
solve the previous ones but the core
problem to me was exploration and
solving it has been basically kind of
the focus and how we saw the first
breakthroughs so exploration you know in
a multi hierarchical way so like 22
times the second exploration is a very
different meaning than it does in terms
of should I gather resources early or
should I wait or so on so how do you
solve the long-term let's talk about the
internals of alpha stuff so first of all
how do you represent the state of the
game as an input right how do you then
do the long term sequence modeling how
do you build a policy right also what's
the architecture like so alpha star has
obviously several components but
everything passes through what we call
the policy which is a neural network and
that's kind of the beauty of it there is
I could just now give you a neural
network and some weights
and if you fed the right observations
and you understood the actions the same
way we do you would have basically the
agent playing the game there's
absolutely nothing else needed other
than those weights that were trained now
the first step is observing the game and
we've experimented with a few
alternatives the one that we currently
use mixes both spatial sort of images
that you would process from the game
that is the zoomed out version of the of
the map and also assume the inversion of
the camera or the screen as we call it
but also we give to the agent the list
of units that it sees more of as a set
of objects that it can operate on that
is not necessarily required to use it
and we have versions of the game that
play well without this set vision that
is a bit not like how humans perceive
the game but it certainly helps a lot
because it's a very natural way to
encode the game is by just looking at
all the units that there are there they
have properties like health position
type of unit whether it's my unit or the
enemies and that's sort of is kind of
the the summary of the state of the of
the game note that list of units or set
of units that you see all the time well
that's pretty close to the way humans
see it again why do you say it's not
isn't it you're saying the exactness of
it is not yeah other humans the
exactness of it is perhaps not the
problem I guess maybe the problem if you
look at it from how actually humans play
the game is that they play with a mouse
on a keyboard and a screen and they
don't see sort of a structured object
with all the units what they see is what
they see on the screen right yes so
remember that there's a certain
interrupt there's a plot that you showed
with camera base where you do exactly
that right move around and that seems to
converge it to similar performance yeah
I think that's what I we're kind of
experimenting with what's necessary or
not but using the set so actually if you
look at research in computer vision
where it makes a lot of sense to treat
images as two-dimensional arrays there
is actually a very nice paper from
Facebook I think I forgot who the
authors
but I think it's part of gaming's has
group and what they do is they take an
image which is this two-dimensional
signal and they actually take pixel by
pixel and scramble the image as if it
was just a list of pixels and crucially
they encode the position of the pixels
with at the XY coordinates and this is
just kind of a new architecture which we
incidentally also use in StarCraft
called the transformer which is a very
popular paper from last year which
yielded very nice result in machine
translation and if you actually believe
in this kind of or it's actually a set
of pixels as long as you encode X Y it's
okay
then you you could argue that the list
of units that we see is precisely that
because we have each unit as a kind of
pixel if you will and then there XY
coordinates so in that perspective we
without knowing it we use the same
architecture that was shown to work very
well on Pascal on image net and so on so
the interesting thing here is putting it
in that way it starts to move it towards
the way you usually work with language
so what and especially with your
expertise and work in language it seems
like there's echoes of a lot of the way
you would work with natural language in
the way you've approached alpha star
right what's does that help with the
long term sequence modeling there
somehow exactly so so now that we
understand what an observation for a
given time step is we need to move on to
say well there's going to be a sequence
of such observations and an agent will
need to given all that it's seen not
only the current time step but all that
it's seen why because there is partial
observability we must remember whether
we saw a worker going somewhere for
instance right because then there might
be an expansion and the top right of the
map so given that what you must then
think about is there is the problem of
given all the observations you have to
predict the next action and not only
given all the observations but given all
the observations and given all the
actions you've taken predict the next
action and that sounds exactly like
machine translation where and that's
exactly how kind of I saw the problem
especially when you are given supervised
data or replays from humans because the
problem is exactly the same you're
translating essentially a prefix of
observations and actions onto what's
going to happen next which is exactly
how you would train a model to translate
or to generate language as well right
you have a certain prefix you must
remember everything that comes in the
past because otherwise you might start
having noncoherent text and the same
architectures we're using LST MS and
transformers to operate on across time
to kind of integrate all that's
happening in the past those
architectures that work so well in
translation or language modeling are
exactly the same than what the agent is
using to issue actions in the game and
the way we train in moreover for
imitation which is step one of alpha
studies take all the human experience
and try to imitate it much like you try
to imitate translators that translated
many pairs of sentences from French to
English say that sort of principle
applies exactly the same it's you
mightyou it's almost the same code
except that instead of words you have a
slightly more complicated objects which
are the observations and the actions are
also a bit more complicated that than
award is there a self play component
into so once you run out of imitation
right so so indeed you can bootstrap
from human replays but then the agents
you get are actually not as good as the
humans you imitated right so how do you
imitate well we take humans from 3,000
MMR and hire 3,000 MMR is just a metric
of human skill and 3,000 MMR might be
like 50% percentile right so it's just
Kevin average human what's that so maybe
quick pause MMR's ranking scale the
matchmaking rating yeah for players so
street uh remember there's like a master
and a grandmaster with 3000 so 3,000 is
is pretty bad I think it's kind of gold
level it just sounds really good
relative to chess I think oh yeah I know
the the rating is the best in the world
are at 7,000 mm
so 3,000 it's a bit like Eloy indeed
right so 3,300 just allows us to not
filter a lot of the data so we like to
have a lot of data in deep learning as
you probably know so we take these kind
of 3,500 and above but then we do a very
interesting trick which is we tell the
neural network what level they are
imitating so we say these replay you're
gonna try to imitate to predict the next
action for all the actions that you're
gonna see is a 4,000 mm our replay this
one is a 6,000 mm our replay and what
what's cool about this is then we take
this policy that is being trained from
human and then we can ask it to play
like a 3,000 mm our player by setting a
bit saying well okay play like a 3,000
mm our player or play like a 6,000 mm
our player and you actually see how the
policy behaves differently
it gets worse economy if you play like a
gold level player it does less actions
per minute which is the number of clicks
or number of actions that you will issue
in a whole minute and it's very
interesting to see that it kind of
imitates the skill level quite well but
if we ask you to play like a 6000 mm our
player
we tested of course these policies to
see how well they do they actually beat
all the built-in ai's that these are put
in the game but they're nowhere near
6000 mm our players right they might be
maybe around gold level
platinum perhaps so there's still a lot
of work to be done for the policy to
truly understand what it means to win so
far we only asked them ok here is the
screen and that's what happened on the
game until this point what would the
next action be that we asked you know we
asked a pro - now say all this you're
gonna click here or here or there and
the point is experiencing experiencing
wins and losses is very important to
then start to refine otherwise the
policy can get loose can can just go off
policy as we call it that's so
interesting you can at least hope
eventually to be able to control a
policy approximately to be a some MMR
level that's that's so interesting
especially given that you have ground
truth for a lot of these cases right can
ask your personal questions or what's
your mmm
well I haven't played Starcraft 2 so I
am unranked oh is the kind of Lois
League okay so I used to play Starcraft
the first one and but you haven't
seriously played so the best player we
have a deep mind is about five thousand
MMR which is high masters is not at
Grandmaster level
Grandmaster level would be the top 200
players in a certain region like Europe
or America or Asia but for me it would
be hard to say I am very bad at the game
I actually played alpha star a bit too
late and it bit me I remember the whole
team was Oreo you should play yeah and I
was it looks like it's not so good yet
and then I remember I kind of got busy
and waited an extra week and I played
and it really beat me
very badly was it I've heard that feel
it's not an amazing feelings amazing
yeah I mean obviously I tried my best
and I try to also impress my because I
actually played the first game so I'm
still pretty good at micromanagement
um the promise I just don't understand
Starcraft 2 I understand Starcraft and
when I played Starcraft I probably was
consistently like for for a couple years
top 32 in Europe so I was decent but at
the time we didn't have this kind of MMR
system as as well established so it
would be hard to know what what it was
back then so what's the difference in
interface between alpha star and
Starcraft and a human player and
Starcraft is there any significant
differences between the way they both
see the game I would say the way they
see the game there's a few things that
are just very hard to simulate the main
one perhaps which is obvious in
hindsight is what's called cloaked units
which are invisible units so in
StarCraft you can make some units that
you need to have a particular kind of
unit to detect it so these units are
invisible
if you cannot detect them you cannot
target them so they would just you know
destroy your buildings or kill your
workers but
despite the fact you cannot target the
unit there's a shimmer that as a human
you observe I mean you need to train a
little bit you need to pay attention but
you would see this kind of space-time as
space-time like distortion and you
wouldn't know okay there art yeah yeah
it's like a wave thing yeah she's kinda
storsch I don't like it that's really
like the Blizzard term is shimmer
shimmer and so this shimmer professional
players actually can see it immediately
they understand it very well but it's
still something that requires certain
amount of attention and and and it's
kind of a bit annoying to deal with
whereas four alpha star in terms of
vision it's very hard for us to simulate
sort of
you know are you looking at these pixel
in the screen and so on so um the only
thing we can do is we there is a unit
that's invisible over there so alpha
star would know that immediately
obviously still obeys the rules you
cannot attack the unit you must have a
detector and so on but it's it's kind of
one of the main things that it just
doesn't feel there's there's a very
proper way I mean you could imagine are
you you don't have high present know
exactly where it is or sometimes you see
it sometimes you don't but it's it's
just really really complicated to get it
so that everyone would agree oh that's
that's the best way to simulate this
right you know it seems like a
perception problem it is a perception
problem so so the only problem is people
are you ask what's the difference
between how humans perceive the game I
would say they wouldn't be able to tell
a shimmer immediately as it appears on
the screen
whereas alpha star in principal sees it
very sharply right it seems okay it sees
that the beat turned from zero to one
meaning there's now a unit there
although you don't know the unit or you
don't know it you know you know that you
cannot attack it and so on God so that
from from a vision standpoint that
probably is the one that is kind of the
most obvious one then there are things
humans cannot do perfectly even
professionals which is they might miss a
detail or they might have not seen a
unit and obviously as a computer if
there's a corner of the screen that
turns green because a unit enters the
field of view that can go into the
memory of the agent the lsdm and Percy's
there for a while
for whatever for however long is
relevant right and in terms of action it
seems like the rate of action from an
alpha star is comparative if not slower
than professional players but is there
but it's more precise as well right so
so that's that's a very like that's
really probably the one that is causing
us more issues for a couple of reasons
right
the first one is Starcraft has been an
AI environment for quite a few years in
fact I mean I was participating in the
very first competition back in 2010 and
there's really not been a kind of a very
clear set of rules how the actions per
minute the rate of actions that you can
issue is and as a result these agents or
bots that people build in a kind of
almost very cool way they do like 20,000
40,000 actions per minute now now to put
this in perspective a very good
professional human my du 300 to 800
actions per minute they might not be as
precise that's why the range is a bit
tricky to to identify exactly I mean 300
actions per minute precisely is probably
realistic 800 is probably not but you
see humans doing a lot of actions
because they warm up and they kind of
select things and spam and so on just so
that when they need they have the
accuracy so we came into this by not
having kind of a standard way to say
well how do we measure whether an agent
is at human level or not on the other
hand we had a huge advantage which is
because we do imitation learning agents
turned out to act like humans in terms
of rate of actions even Precision's and
imprecisions of actions in the supervise
policy you could see all this you could
see how agents like to spam click to
move here if you played specially Diablo
you would know what I mean I mean you
just duck like spam or a move here
Murphy a move here you're doing
literally like maybe five actions in two
seconds but these actions are not very
meaning meaningful one would have
sufficed so on the one hand we start
from this imitation policy that is at
the ballpark of the actions per millions
of humans because it acts
statistically trying to imitate humans
so we see these very nicely in the
curves that we showed in the blog post
like this these actions per minute and
the distribution looks very human-like
but then of course as self play kicks in
and and that's the part we haven't
talked too much yet but of course the
agent must played getting itself to
improve then there's almost no
guarantees that these actions will not
become more precise or even the rate of
actions is going to increase over them
so what we did and this is probably kind
of the first attempt that we thought was
reasonable is we looked at the
distribution of actions for humans for
certain windows of time and just to give
a perspective because I guess I
mentioned that some of these agents that
are programmatic let's call them they do
40,000 actions per minute professionals
as I said to 300 to 800 so what we
looked is we look at a distribution over
professional gamers and we took
reasonably high actions per minute but
we kind of identify certain cat offs
after which if even if the agent wanted
to act these actions would be dropped
but the problem is this cutoff is
probably set a bit too high and what
ends up happening even though the games
and when we ask the professionals and
the gamers by by and large they feel
like it's playing human-like there are
some Asians that developed maybe
slightly too high a PMS which is actions
per minute combined with the precision
which made people sort of start
discussing a very interesting issue
which is should we have limited these
should we just let it loose and see what
cool things it can come up with right so
this is in itself an extremely
interesting question but the same way
that modeling the shimmer would be so
difficult
modeling absolutely all the details
about muscles and precision and an
tiredness of humans would be quite
difficult right so we're here in kind of
innovating in this sense of okay what
could be maybe the next iteration of
putting more rules that makes the agents
more human-like in terms of restrictions
yeah putting constraints that more
constraints yeah that's really
interesting that's really innovative so
one of the constraints you put on your
on yourself or at least focused in is on
the protoss race as far as I understand
can you tell me about the different
races and how they so protoss terran and
the Zerg how do they compare how do they
interact why did you choose Protoss in
right there is in the dynamics of the
game you seen from a strategic
perspective so Protoss so in stacked of
there are three races indeed in the
demonstration we saw only the Protoss
race so maybe let's start with that one
Protoss is kind of the most
technologically advanced race it has
units that are expensive but powerful
right so in general you want to kind of
conserve your units as you go attack so
you wanna and and then you want to
utilize these tactical advantages of
very fancy spells and so on so forth and
at the same time they're kind of people
say like they're they're a bit easier to
play perhaps right but that I actually
didn't know I mean I just talked to now
a lot to the players that we we work
with TLO and mana and they said oh yeah
Protoss is actually people think is
actually one of the easiest races so
perhaps the easier that doesn't mean
that it's you know obviously
professional players excel at the three
races and there's never like a race that
dominates for a very long time anyway so
if you look at the top and 100 in the
world is there one race that dominates
that list it would be hard to know
because it depends on the regions I
think it's pretty equal in terms of
distribution and Blizzard wants it to be
equal right they don't want they
wouldn't want one race like Protoss to
not be representative in the top place
so definitely like they tried it to be
like the balance right so then maybe the
opposite race of Protoss is zerk dirt is
a race where you just kind of expand and
take over as many resources as you can
and they have a very high capacity to
regenerate their units so if you have an
army it's not that valuable in terms of
losing the whole army is not a big deal
azarkh because you can then rebuild it
and given that you generally accumulate
a huge Bank of resources zurk steep it
will play by a prayer applying a lot of
pressure may be losing their whole army
but then rebuilding it quickly so
although of course every race I mean
there is never I mean they're pretty
diverse I mean there's some unity insert
that are technologically advanced and
they do some very interesting spells and
there's some units in proto's that are
less valuable and you could lose a lot
of them and rebuild them and it wouldn't
be a big deal all right so maybe I'm
missing out maybe I'm gonna say some
dumb stuff but just summary of strategy
so first there's collection of a lot of
resources right so that's one option the
other one is expanding so building other
basins then the other is obviously
attack ability building units and
attacking with those units and then I
don't know what else there is maybe
there is the different timing of attacks
like to attack early attack ready what
are the different strategies that
emerged that you've learned about I've
read the a bunch of people are super
happy that you guys have apparently that
alpha star apparently is discovered
that's really good too what is it
saturate oh yeah they're mind-numbing
online yeah the mineralogist yeah yeah
the in it for greedy amateur players
like myself that's always been a good
strategy you just build up a lot of
money and it just feels good it's just
accumulate and accumulate so thank you
for discovering that yeah validating all
of us but is there other strategies that
you discovered interesting yeah unique
to to this game yeah so if you look at
the kind of not being a stack of two
player but of course Starcraft and
Starcraft 2 and realtime strategy games
in general are very similar I would
classify perhaps the openings of the
game they're very important and
generally I would say there's two kinds
of openings one that's a standard
opening that's generally how players
find
sort of a balance between risk and
economy and building some you needs
early on so that they could defend but
they're not to expose basically but also
expanding quite quickly so this is would
be kind of a standard opening and within
a standard opening then you what you do
choose general is what technology are
you aiming towards so there's a bit of
rock-paper-scissors of you could go for
spaceships or you could go for invisible
units or you could go for I don't know
like massive units that attack against
certain kinds of units but they're weak
against others so standard openings
themselves have some choices like
rock-paper-scissors style of course if
you Scout and you're good at guessing
what the opponent is doing then you can
plane as an advantage because if you
know you're gonna play rock I mean I'm
gonna play paper obviously so you can
imagine that normal standard games in
StarCraft looks like a continuous rock
paper scissor game where you guess what
the distribution of rock paper and
scissor is from the enemy and reacting
accordingly to try to beat it or you
know put the paper out before he kind of
changes his mind from rock to scissors
and then you would be in a weak position
so sorry to pause on that I didn't
realize this element cuz I know it's
true but poker I know I looked at
Lambrakis
you're this is so you're also estimating
trying to guess the distribution to
better and better estimate the
distribution what the Epona's likely to
be doing yeah I mean as a player you
definitely want to have a belief state
over what's up on the other side of the
map and when your belief state becomes
inaccurate when you start having that
serious doubts whether he's gonna play
something that you must know that's when
you scout you wanna then gather
information right it's improving the
accuracy of the belief or improving the
belief state part of the loss that you
try to optimize or is it just in a side
effect it's implicit but you could
explicitly model it and it would be
quite good that's probably predicting
what's on the other side of the map but
so far it's all implicit the lot there's
no no additional reward for predicting
the enemy so there's these standard
opening x' and then there's what people
call
which is very interesting and alpha star
sometimes really likes this kind of
cheese these cheese's what they are is
kind of an all-in strategy you're gonna
do something sneaky you're gonna hide
enemies as hide your own buildings close
to the enemy base or you're gonna go for
hiding your technological buildings so
that you do invisible units and the
enemy just cannot react to detect it and
does lose the game and there's quite a
few of these cheeses and variants of
them and there is where actually the
belief state becomes even more important
because if I spelled your base and I see
no buildings at all any human prayer
knows something's up they might know
well you're hiding something close to my
base should I build suddenly a lot of
units to defend should I actually block
my ramp with workers so that you cannot
come and destroy my base so there's all
these is happening and defending against
Jesus is extremely important and in the
alpha star League many agents actually
develop some cheesy strategies and in
the games we saw against elo and mana
two out of the ten agents were actually
doing these kind of strategies which are
cheesy strategies and then there's a
baron of cheesy strategy which is called
all-in so an all-in strategy is not
perhaps as drastic as oh I'm gonna build
cannons on your base and then bring all
my workers and try to just disrupt your
base and game over or GG as we say in
StarCraft umm there's these kind of very
cool things that you can align precisely
at a certain time mark so for instance
you can generate exactly ten unit
composition that is perfectly five of
these type five of disorder type and
align the upgrade so that at four
minutes and a half let's say you have
these ten units and the upgrade just
finished and at that point that army is
really scary and unless the enemy really
knows what's going on
if you push you might then have an
advantage because maybe the enemy is
doing something more standard it
expanded too much it developed too much
economy and any trade-off badly against
having defenses and the enemy will lose
but it's called all-in because if you
don't win then you're gonna lose
so you see players that do these kind of
strategies if they don't succeed game is
not over I mean they still have a place
and they still gathering minerals but
they will just gg out of the game
because they know well game is over I
gambled and I failed
so if we start entering the game
theoretic aspects of the game it's
really rich and it's really that's why
it also makes it quite entertaining to
watch even if I don't play I still enjoy
watching the game but the agents are
trying to do this mostly implicitly but
one element that we improved in self
plays creating the alpha star League and
the alpha star League is not pure self
play it's trying to create different
personalities of agents so that some of
them will become cheese cheese agents
some of them might become very
economical very greedy like getting all
the resources but then being maybe early
on they're gonna be weak but later on
they're gonna be very strong and by
creating this personality of agents
which sometimes it just happens
naturally that you can see kind of an
evolution of agents that given the
previous generation they trained against
all of them and then they generate kind
of the count the perfect counter to that
distribution but these these agents you
must have them in the populations
because if you don't have them you're
not covered against these things right
it's kind of you wanna you wanna you
know create all sorts of the opponents
that you will find in the wild so you
can be exposed to these cheeses early
aggression later aggression more
expansions dropping units in your base
from the side all these things and pure
self play is getting a bit stack at
finding some subset of these but not all
of these so the alpha star League is a
way to kind of do an example of agents
that they're all playing and in a league
much like people play on battlenet right
they play you play against someone who
does a new cool strategy and you
immediately oh my god I want to try it I
want to play again and these to me was
another critical part of the of the of
the problem which was can we create a
battlenet for agents yeah that's kind of
what the Alpha star league is really
fascinating and where they stick to
their different strategies yeah wow
that's
it's really really interesting so but
that said you were fortunate enough or
just skilled enough to win 5-0 and so
how hard is it to win I mean that's not
the goal I guess I don't know what the
goal is the goal should be to a majority
not five zero but how hard is it in
general to win all matchups
I don't want 1v1 so that's a very
interesting question because once you
see alpha star and superficially you
think well locate one
let's if you some of the games like ten
to one right it lost the game that it
played with the camera interface you
might think well that's that's done
right there's it's it's super human at
the game and that's not really the claim
we really can make actually the claim is
we beat a professional gamer for the
first time Starcraft has really been a
thing that it's been going on for a few
years but moment a moment like this
hasn't not had not occurred before yet
but our decisions impossible to beat
absolutely not right so that's a bit
what's you know kind of that the
difference is the agents play at
Grandmaster level there definitely
understand the game enough to play
extremely well but are they am beatable
do they play perfect no and actually in
StarCraft because of these sneaky
strategies it's always possible that you
might take a huge risk sometimes but you
might get wins right out of out of this
so I think that as a domain it still has
a lot of opportunities not only because
of course we want to learn with less
experience we would like to I mean if I
if I learn to play Protoss I can play
turn and learn it much quicker than
alpha star can right so there are
obvious interesting research challenges
as well but even as as the raw like as
the raw performance goes really the
claim here can be we are at pro level or
at high Grandmaster level but obviously
the players also did not know what to
expect right this kind of their prior
distribution was a bit off because they
played this kind of new like alien brain
as they like to say drive and
that's what makes it exciting for them
but also I think if you look at the
games closely you see there were
weaknesses in some points maybe alpha
star did not scout or if it had got
invisible units going against at certain
points it wouldn't have known and it
would have been bad so there's still
quite a lot of work to do but it's
really a very exciting moment for us to
be seeing Wow a single neural net on a
GPU is actually playing against these
guys who are amazing I mean you have to
see them play in life if they're really
really amazing players yeah I'm sure
there's there's the most there must be a
guy in Poland somewhere right now
training his butt off to make sure that
this never happens again with alpha star
so that's really exciting in terms of
alpha star having some holes to exploit
yeah it's just great and then you build
on top of each other and it feels like
Starcraft I'll like go even if you win
it's still not there it's still not
there's so many different dimensions in
which you can explore so that's really
really interesting do you think there's
a ceiling to alpha star you've said that
it hasn't reached you know it's this is
a big wait what you know let me actually
just pause for a second how did it feel
to come here to this point to beat a top
professional player like that night I
mean you know Olympic athletes have
their gold medal right this is your gold
medal and sense sure you're cited a lot
you published a lot of prestige papers
whatever but this is like a win
how did it feel I mean it was for me was
unbelievable because first the win
itself me was so exciting I mean d so
looking back to those last days of 2018
really well that's when the games were
played I'm sure I look back at that
moment I say oh my god I want to be it
like in a project like that it's like I
already feel the nostalgia of like yeah
that was huge in terms of the energy and
the team effort that went into it and so
in that sense as soon as it happened I
already knew it was kind of I was losing
it a little bit so it is almost like sad
that it happened and all
like but on the other hand it also
verifies the approach but to me also
there's so many challenges and
interesting aspects of intelligence that
even though we can train a neural
network to play at the level of the best
humans there's still so many challenges
so for me it's also like well this is
really an amazing achievement but I
already was also thinking about next
steps I mean as I said these agents play
Protoss vs. Protoss but they should be
able to play a different race much
quicker right so that would be an
amazing achievement some people call
this matter reinforcement learning meta
learning and so on right so there's so
many possibilities after that moment but
the moment itself it really felt great
it's I we had this bet so I I'm kind of
a pessimist in general so I kind of send
an email to the team and said okay let's
again steal offers right like what's
gonna be the result and I really thought
we would lose like five zero right III
we had some calibration made against the
5000 MMR player TLO was much stronger
than that player even if he played
Protoss which is his off race but yeah I
was not imagining we would win so for me
that was just kind of a test run or
something and then it really kind of he
was really surprised and unbelievably we
went to this to this bar to celebrate
and and dave tells me well why don't we
invite someone who is a thousand mm are
stronger in proto's like an actual
Protoss player like like that it turned
up being man all right and you know we
had some drinks and I said sure why not
but then I thought well that's really
gonna be impossible to beat I mean even
because it's so much I had a thousand
MMR is really like 99% probability that
mana would beat TLO as Protoss vs.
Protoss right so we did that and to me
the second the second game was much more
important even though a lot of
uncertainty kind of disappeared after we
we cannot beat the yellow I mean it he
is a professional player so that was
kind of over that's really a very nice
achievement
but mana really was at the top and you
could see he played much better but our
agents got much better too so it's a and
then after the first game I said if we
take a single game at least we can say
we beat a game I mean even if we don't
beat the series for me that was a huge
relief and I mean I remember hugging
them is and I mean it was it was really
like this moment for me will resonate
forever as a researcher and I mean as a
person and is a really like great
accomplishment and it was great also to
be there with the team in the room I
don't know if you saw like so it was
really like I mean from my perspective
the other interesting thing is just like
watching Kasparov watching mana was also
interesting because he didn't he is kind
of a loss of words I mean whenever you
lose I've done a lot of sports you
sometimes say excuses you look for
reasons right and he couldn't really
come up with a reason yeah yeah I mean
so with the off race for Protoss you
could say it was it felt awkward it
wasn't but here it was yeah it was it
was just beaten and it was beautiful to
look at a human being being superseded
by an AI system I mean it's a it's a
beautiful moment for researchers so yeah
for sure it was it was I mean probably
the highlight of my career so far
because of its uniqueness and coolness
and I don't know I mean it's obviously
as you said you can look at paper
citations and so on but these these
really is like a testament of the whole
machine learning approach and using
games to advance technology I mean it's
really it really was everything came
together at that moment that that's
really the summary also on the other
side it's a popularization of AI too
because just like traveling to to the
moon and so on I mean this is where a
very large community of people that
don't really and no way I get to really
interact with it which is very important
I mean it's really we must you know
writing papers helps our peers
researchers to understand what we're
doing but I think AI is becoming mature
enough that we must sort of try to
explain what it is and perhaps through
games is an obvious way because these
games always had built
so it may be everyone experience an AI
playing a video game even if they don't
know because there's always some
scripted element and some people might
even call that AI already right so what
are other applications of the approaches
underlying alpha star that you see
happening there's a lot of echoes of he
said transformer of language modeling so
on have you already started thinking
where the breakthroughs in alpha star
get expanded to other applications right
so I thought about a few things for like
kind of next months next year's the main
thing I'm thinking about actually is
what's next as a kind of a grand
challenge because for me like we've seen
Atari and then there's like the sort of
three-dimensional worlds that we've seen
also like pretty good performance from
these capture-the-flag
agents that also some people at deep
mine and elsewhere are working on we've
also seen some amazing results on like
for instance dota 2 which is also a very
complicated game so for me like the main
thing I'm thinking about is what's next
in terms of challenge so as a researcher
I see sort of two tensions between
research and then applications or areas
or domains where you apply them so on
the one hand we've done thanks to the
application of StarCraft is very hard we
develop some techniques some new
research that now we could look at
elsewhere like are there other
applications where we can apply this and
the obvious ones absolutely you can
think of feeding back to sort of the
community we took from which was mostly
sequence modeling or natural language
processing so we've developed an
extended things from the transformer and
and we use pointer networks we combine
LSD and transformers in interesting ways
so that perhaps the kind of lowest
hanging fruit of feeding back to now
different fields of machine learning
that's not playing video games let me go
old-school and jump to the
to mr. Alan Turing yeah so the Turing
test you know there's a natural language
test the conversational test what's your
thought of it as a test for intelligence
do you think it is a grand challenge
that's worthy of undertaking maybe if it
is would you reformulate it or phrase it
somehow differently right so I really
love the Turing test because I also like
sequences and language understanding and
in fact some of the early work we did in
machine translation we tried to apply to
apply to kind of a neural chat bot which
obviously would never pass the Turing
test because it was very limited but it
is a very fascinating fascinating idea
that you could really have an AI that
would be indistinguishable from humans
in terms of asking or conversing with
with it right so I think the test itself
seems very nice and it's kind of well
defined actually like the passing it or
not I think there's quite a few rules
that feel like pretty simple and and and
you know you could you could really like
have I mean I think they have these
competitions every year yes of the
laudner prize but I don't know if you've
seen a I don't know if you've seen the
kind of bots that emerge from that
competition they're not quite as what
you would so it feels like that there's
weaknesses with the way tauren
formulated it it needs to be that the
definition of a genuine rich fulfilling
human conversation it needs to be
something else like the Alexa prize
which I'm not as well familiar with has
tried to define that more I think by
saying you have to continue keeping a
conversation for 30 minutes something
like that
so basically forcing the agent not to
just fool but to have an engaging
conversation kind of thing is that I
mean is is this have you thought about
this problem richly like as and if if
you have in general how how far away are
we from you worked a lot on language
understanding language generation but
the full dialogue the conversation you
know just sitting at the bar having a
couple of beers for an hour and that
kind of conversation have you thought
about yeah so I think you touched here
on the critical point which is
feasibility right so so there's there's
a great sort of essay by Hamming which
describes sort of grand challenges of
physics and he argues that well okay for
instance teleportation or time travel
our great grand challenges of physics
but there's no attacks we really don't
know or cannot kind of make any progress
so that's why most physicists and so on
they don't work on these in their PhDs
and and as part of their careers so I
see the Turing test as I in the full
Turing test as a bit still too early
like I am I think we're especially with
the current trend of deep learning
language models we've seen some amazing
examples I think GPD to being the most
recent one which is very impressive but
to understand to fully solve passing or
fooling a human to think that you're
that there's a human on the other side I
think we're quite far so as a result I
don't see myself and I probably would
not recommend people doing a PhD on
solving the Turing test because it just
feels it's kind of too early or too hard
of a problem yeah but that said you said
the exact same thing about Starcraft
about a few years ago so into damage so
I prefer yeah you'll probably also be
the person who passes the Turing test in
three years I mean I think I think that
yeah so so we have the Sun record this
is nice it's really I mean that the it's
true that progress sometimes is a bit
unpredictable I really wouldn't have not
even six months ago I would not have
predicted the level that we see that
these agents can deliver at Grandmaster
level but I I have worked on language
enough and basically my concern is not
that something could happen a
breakthrough could happen that would
bring us to solving or passing the
Turing test is that I just think the
statistical approach to it like this it
is not is not gonna cut it so we need we
need the breakthrough we
is great for the community but given
that I think there's quite a more
uncertainty whereas for StarCraft I knew
what the steps would be to kind of get
us there I think it was clear that using
the imitation learning part and then
using these battlenet for agents were
gonna be key and and it turned out that
this was the case and a little more was
needed but not much more for Turing test
I just don't know what the plan or
execution plan would look like so that's
why I'm I myself working on it as a
grand challenge is hard but there are
quite a few sub challenges that are
related that you could say well I mean
what if you create a great assistant
like Google already has like the Google
assistant so can we make it better and
can we make it fully new role and so on
that I start to believe maybe we're
reaching a point where we should attempt
these challenges like this conversation
so much because the echo is very much to
start a conversation it's exactly how
you approach StarCraft let's break it
down into small pieces solve those and
you end up solving the whole game great
but that said you you're behind some of
the sort of biggest pieces of work and
deep learning in the last several years
so you mentioned some limits what do you
think of the current limits of deep
learning and how do we overcome those
limits so if I had to actually use a
single word to define the main challenge
in deep learning is a challenge that
probably has been the challenge for many
years and is that of generalization so
what that means is that all that we're
doing is fitting functions to data and
when the data we see is not from the
same distribution or even if there some
times that it is very close to
distribution but because of the way we
train it with limited samples we then
get to this stage where we just don't
see generalization as much as we can
generalize and I think adversarial
examples are a clear example of these
but if you study machine learning and
literature and you know the reason why
as VMs came very popular where because
they were dealing and they
had some guarantees about generalization
which is unseen data or out of
distribution or even within distribution
where you take an image adding a bit of
noise
these models fail so I think really I
don't see a lot of progress on
generalization in in the strong
generalization sense of the word I I
think our neuron neural networks you can
always find design examples that will
make their outputs arbitrary which is
which which is not good because we
humans would never be fooled by these
kind of images or manipulation of the
image and if you look at the mathematics
you kind of understand this is a bunch
of matrices multiplied together
there's probably numerix and instability
that you can just find corner cases so I
think that's really the underlying topic
many times we see when even even at the
grand stage of like doing test
generalization I mean if used if you
start
I mean passing the Turing test should
you should it be in English or should it
be in any language right I mean as a
human if you could you could if you ask
something in a different language you
actually will go and do some research
and try to translate it and so on
shoot the Turing test in clementa
include that right and it's really a
difficult problem and very fascinating
and very mysterious actually yeah
absolutely but do you think it's if you
were to try to solve it can you not grow
the size of data intelligently in such a
way that the distribution of your
training set does include the entirety
of the testing set I think is that one
path the other path is totally new
methodology right it's not statistical
so a path that has worked well and it
worked well in in stark Ravin in machine
translation and in languages scaling up
the data and the model and that's kind
of been maybe the only single formula
that the leap still delivers today in
deep learning right it's it's that scale
data scale and model scale really do
more and more of the things that we
thought oh there's no way it can
generalize to these Ori there's no way
it can generalize to that but I don't
think fundamentally it
resolve with these and for instance I'm
really liking some style or approach
that would not only have neural networks
but it would have programs or some
discrete decision-making because there
is what I feel there's a bit more like
like I mean the example of the best
example I think for understanding
disease I also worked a bit on or like
we can learn an algorithm with a neural
network right so you give it many
examples and it's going to sort your
sort the input numbers or something like
that but really strong generalization is
you give me some numbers or you ask me
to create an algorithm that sorts
numbers and instead of creating a neural
net which will be fragile because it's
gonna go out of range at some point
you're gonna give you numbers that are
too large to small and whatnot
you just if you just create a piece of
code that sorts the numbers then you can
prove that that will generalize to
absolutely all the possible inputs you
could give so I think that's the problem
comes with some exciting prospects I
mean scale is a bit more boring but it
really works and then maybe programs and
these critics tractions are a bit less
developed but clearly I think they're
quite exciting in terms of future for
the field do you draw any insight wisdom
from the 80s and expert systems and
symbolic systems about computing do you
ever go back to those reasoning that
kind of logic do you think that might
make a comeback you have to dust off
those books yeah I actually love
actually adding more inductive biases to
me the problem really is what are you
trying to solve if what you're trying to
solve is so important that try to solve
it no matter what then absolutely use
rules use domain knowledge and then use
a bit of the magic of machine learning
to empower to make the system as the
best system that will detect cancer or
you know or detect weather patterns
right or in terms of start of it also
was a very big challenge so I was
definitely happy that if we had to get
take cut a corner here and there it
could have been
interesting to do and in fact in
StarCraft we we start thinking about
expert systems because it's very you
know you can define I mean people
actually build stack reports by thinking
about those principal I guess you know
state machines and rule-based and then
you could you could think of combining a
bit of a rule-based system but that has
also neural networks incorporated to
make it generalize a bit better so
absolutely I mean we should we should
definitely go back to those ideas and
anything that makes the problem simpler
as long as your problem is important
that's okay and that's research driving
a very important problem and on the
other hand if you wanna really focus on
the limits of reinforcement learning
then of course you must try not to look
at imitation data or to look some like
for some rules of the domain that would
help a lot or even feature engineering
right so these these attention that
depending on what you do I think both
both ways are definitely fine and I
would never not do one or the other if
you're as long as you what you're doing
is important and needs to be soft right
right so there's a bunch of different
ideas that that that you develop that I
really enjoy so but one one is
translating from the image captioning
translated finish the text just just
another just beautiful yeah beautiful
idea I think that resonates throughout
your work actually so the underlying
nature of reality being language always
yes somehow so what's the connection
between images and text rather the
visual world and the world of language
in your view right so I think a piece of
research that's been central to I would
say even extending into Starcraft is is
this idea of sequence to sequence
learning which what we really meant by
that is that you can you can now really
input anything to a neural network as
the input X and then the neural network
will learn a function f that will take X
as an input and produce any output Y and
these x and y's don't need to be like
static or like a features like as
like a fixed vectors or anything like
that it could be it really sequences and
now beyond like data structures right so
that paradigm was tested in a very
interesting way when we moved from
translating French to English to
translating an image to its caption but
the beauty the beauty of it is that
really and that's actually how it
happened I run I change the line of code
in this thing that was doing machine
translation I and I came the next day
and I saw how it like it was producing
captions that seemed like oh my god this
is really really working and the
principle is the same right so I think I
don't see text vision speech waveforms
as something different here as long as
you basically learn a function that will
vector eyes you know these into and then
after we vectorize it we can then use
you know transformers LS DMS whatever
the flavor of the month of the model is
and then as long as we have enough
supervised data really this formula will
work and we'll keep working I believe to
some extent model of these
generalization issues that I mentioned
before so but the testers to vectorize
sort of former representation that's
meaningful nothing and your intuition
now having worked with all this media is
that once you are able to form that
representation you can basically take
any things any sequence is there go back
to Starcraft is there limits on the
length so we didn't really touch on a
long term effect how did you overcome
the whole really long term aspect of
things here is there some tricks or so
the main streak so Starcraft if you look
at absolutely every frame you might
think it's it's quite a long game so we
would have to multiply 22 times 60
seconds per minute times maybe at least
10 minutes per game on average so there
are quite a few frames but the trick
really was to only observe in fact which
might be seen as a limitation but
it is also computational advantage only
observe when you act and then what the
neural network decides is what is the
gap gonna be until the next action and
if you look at most Starcraft games that
we have in the in the data set that
Blaser provided it turns out that most
games are actually only I mean it is
still a long sequence but it may be like
a thousand to 1,500 actions which if you
start looking at L STM's large LST M's
transformers it's it's not like it's not
that that difficult especially if you
have supervised learning if you had to
do it with reinforcement learning the
credit assignment problem what is it
that in this game that made you win that
would be really difficult but thankfully
because of imitation learning we didn't
kind of have to deal with these directly
although if we had to we tried it and
what happen is you just take all your
workers and attack with them and that
sort of is kind of obvious in retrospect
because you start trying random actions
one of the actions will be a worker that
goes to the enemy base and because it's
self play it's not gonna know how to
defend because it basically doesn't know
almost anything and eventually what you
develop is this take our workers and
attack because the the create assignment
issue in Arad is really really hard I do
believe we could do better and that's
maybe a research challenge for the
future but yeah even even in StarCraft
the sequences are maybe a thousand which
I believe there is within the realm of
what transformers can do yeah I guess
the difference between Starcraft and go
is in go and chest stuff starts
happening right away right so there's
not yeah it's pretty easy to self play
not easy but to sulfa is possible to
develop reasonable strategy as quickly
as opposed to Starcraft meaning go
there's only 400 actions but one action
is what people would call the god action
that would be if you had expanded the
whole search tree that's the best action
if you did minimax or whatever algorithm
you would do if you had the
computational capacity
but in StarCraft the 400 is miniscule
like a in 400 you don't even like you
you couldn't even click on the pixels
around a unit right so I think the
problem there is in terms of action
space size is way harder so and that
surge is impossible so there's quite a
few challenges indeed that make this
kind of a step step up in terms of
machine learning for humans
maybe they playing Starcraft it seems
more intuitive because it's looks real I
mean you know like the graphics and
everything moves smoothly
whereas I I don't know how to come in go
is a game that I wouldn't really mean to
study it feels quite complicated but for
machines kind of maybe easier reverse
yes
which shows you the gap actually between
deep learning and however the heck our
brains work so you developed a lot of
really interesting ideas it's
interesting to just ask what's the
what's your process of developing new
ideas
do you like brainstorming with others do
you like thinking alone do you like like
was it eating good fellow said he came
up with Gans after a few beers right he
thinks beers are essential yeah coming
up with new ideas we had beers to decide
to play another game game of Starcraft
after a week so it's really similar to
that story actually I explained this in
a in a deep mind retreat and I said this
is the same as the gun story I mean we
were wearing a bar and we decided let's
play again next week and that's what
happened I feel like we're giving the
wrong message to young undergrads yeah
but in general like yeah do you like
brainstorming do you like thinking alone
working stuff out and so I think I think
throughout the years also things changed
right so initially I was very fortunate
to be with great minds like Geoff Hinton
Jeff Dean Ilya sutskever I was really
fortunate to join brain at a very good
time so at that point it ideas I was
just kind of brainstorming with my
colleagues and learned a lot and keep
learning is actually something you
should never stop doing right so
learning implies reading papers and also
these casting ideas with others it's
very hard at some point to not
communicate that being reading
paper forms from someone or actually
discussing right so definitely that
communication aspect needs to be there
whether it's written or oral nowadays
I'm also trying to be a bit more
strategic about what research to do so I
was describing a little bit this sort of
tension between research for the sake of
research and then you have on the other
hand applications that can drive the
research right and honestly the formula
that has worked best for me is just find
a hard problem and then try to see how
research fits into it how it doesn't fit
into it and then you must innovate so I
think machine translation drove sequence
to sequence then maybe like learning
algorithms that had to like
combinatorial algorithms led to pointer
networks Starcraft led to really scaling
a permutation learning and the Alpha
star league so that's been a formula
that I personally like but the other one
is also about it and I seen it succeed a
lot of the times where you just want to
investigate model-based RL as a kind of
a research topic and then you must then
start to think well how are the tests
how are you going to test these ideas
you need to kind of a minimal
environment to try things you need to
read a lot of papers and so on and
that's also very fun to do and something
I've also done quite a few times both at
brain at the mine and obviously as as a
PhD so so I think besides that the ideas
and discussions I think it's important
also because you start sort of guiding
not only your own goals but other
people's goes to the next breakthrough
so you you must really kind of
understand these you know feasibility
also as we were discussing before right
whether whether these domain is ready to
be tackled or not and you don't want to
be too early you obviously don't want to
be too late so it's it's really
interesting and this is a strategic
component of research which I think as a
grad student I just had no idea to you
know I just read papers and discussed
ideas and I think this has been maybe
the major change and I recommend
people kind of feed forward to success
how it looks like and try to backtrack
other than just kind of looking out
these looks cool these looks cool and
then you do a bit of random work which
sometimes you stumble upon some
interesting things but in general it's
it's also good to plan a bit yeah I like
it
especially like your approach I've taken
a really hard problem stepping right in
and then being super skeptical about
yeah being robbed I mean there's a
balance of both right there's a silly
optimism and and a critical sort of
skepticism that's good to balance which
is why it's good to have a team of
people that that balance that you don't
do that on your own you have both
mentors that have seen or you obviously
wanna chat and discuss whether it's the
right time I mean Demi's came in 2014
and he said maybe in a bit we'll do
starcraft and maybe he knew and that's
and I'm just following his lead which is
great because he's he's brilliant right
so these these things are obviously
quite important that you wanna be
surrounded by people who you know are
diverse they they have their knowledge
there's also important too I mean I I've
learned a lot from people who actually
have an idea that I might not think it's
good but if I give them the space to try
it I've been proven wrong many many
times as well so that's that's great
it's I think it's your colleagues are
more important than yourself I think so
sure now let's real quick talk about
another impossible problem AGI right
what do you think it takes to build a
system that's human level intelligence
we talked a little bit about the
Tauranga stark after all these have
echoes of general intelligence but if
you think about just something that you
would sit back and say wow this is
really something that resembles human
level intelligence what do you think it
takes to build that so I find that AGI
oftentimes is maybe not by well defined
so what I'm trying to then come up with
for myself is what would be a result
look like that you would
start to believe that you would have
agents or neural nets that no longer
sort of over feet to a single task right
but actually kind of learn the skill of
learning so to speak and that actually
is a field that I am fascinated by which
is the learning to learn or meta
learning which is about no longer
learning about a single domain so you
can think about the learning algorithm
itself is general right so the same
formula we applied for alpha star or
Starcraft we can now apply to kind of
almost any video game or you could apply
to many other problems and domains but
the algorithm is what's kind of
generalizing but the neural network the
weights those weights are useless even
to play another race right I train a
network to play very well at Protoss vs.
Protoss I need to throw away those
weights if I want to play now
terran vs terran i would need to retrain
a network from scratch with the same
algorithm that's beautiful
but the network itself will not be
useful so I think when I if I see an
approach that can observe or start
solving new problems without the need to
kind of restart the process I think that
to me would be a nice way to define some
form of AGI again I don't know the grand
views like age I mean so it should
Turing test nice or before AGI I mean I
don't know I think I think concretely I
would like to see clearly that meta
learning happened meaning there there is
an architecture or network that as it
sees new new problem or new data it
solves it and to make it kind of a
benchmark it should solve it at the same
speed that we do solve new problems when
I define your new object and you have to
recognize it when I when you start
playing a new game you played all the
times but now you play a new Atari game
well you you're gonna be pretty quickly
pretty good at the game so that's
perhaps what's the domain and what's the
exact benchmark is a bit difficult I
think as a community we might need to do
some work to define it
but I think this first step I could see
it happen relatively soon but then the
whole what a GI means and so on I am a
bit more confused about what I think
people mean different things there's an
emotional psychological level that like
the even the Turing test passing the
Turing test is something that we just
passed judgment on human beings what it
means to be you know as a as a dog in a
GI system yeah like what level what does
it mean right yeah what does it mean but
I like the generalization and maybe as a
community would converge towards a group
of domains that are sufficiently far
away that would be really damn
impressive if we're able to generalize
some perhaps not as close as Protoss and
Zerg but like Wikipedia step be a good
stuff and then a really good step but
then then like Wickham Starcraft 2
Wikipedia yeah I'm back yeah that kind
of thing and that that feels also quite
hard and far but I think there's as long
as you put the benchmark out as we
discovered for instance with image net
then tremendous progress can be had so I
think maybe there's a lack of benchmark
but I'm sure we'll find one and yeah a
community will will then work towards
that and then beyond what a GI might
mean or would imply I really am hopeful
to see basically machine learning or AI
just scaling up and helping you know
people that might not have the resources
to hire an assistant or that they might
not even know what the weather is like
but you know so I think there's in terms
of the impact the positive impact of AI
I think that's maybe what we should also
not lose focus right the research
community building AG I mean that's a
real nice girl but man I think the way
that deep mind puts it is and then use
it to solve everything else right so I
think we should paralyze yeah we
shouldn't forget about all the positive
things that are actually coming out of
it already and I are not going to be
coming out right but
that I know let me ask relative the
popular perception do you have any worry
about the existential threat of
artificial intelligence in the near or
far future that some people have I think
I'm in the near future I'm I'm skeptical
so I hope I'm not wrong but I'm I'm not
concerned but I I appreciate efforts
ongoing efforts and even like whole
research fields on AI safety emerging
and in conferences and so on I think
that's great in the long term I really
hope we just can simply have the
benefits outweigh the potential dangers
I am hopeful for that but also we must
remain vigilant to kind of monitor and
assess whether the trade-offs are are
there and and we have you know enough
also lead time to prevent or to redirect
our efforts if need be right so but I'm
quite I'm quite optimistic about the
technology and definitely more fearful
of other threats in terms of planetary
level at this point but obviously that's
the one I kind of have more like power
on so clearly I do start thinking more
and more about this and it's kind of
it's groaning me actually to to start
reading more about AI safety jeez afield
that so far I have not really
contributed to but maybe there's
something to be done there as well I
think it's really important you know I
would talk about this issue folks but
it's important to ask you and shove it
in your head because you're at the
leading edge of actually what people are
excited about nay I I mean the work with
alpha star it it's arguably at the very
cutting edge of the kind of thing that
people are afraid of and so you speaking
to that fact and that we're actually
quite far away to the kind of thing that
people might be afraid of but it's still
something worthwhile to think about and
it's also good that you're
the you're not as worried and you're
also open to us yeah me Maura
there's two aspects I mean me not being
worried but obviously we should prepare
for for for it right for for like
forever for things that could go wrong
misuse of the technologies as with any
technologies right so I think there's
there's always trade-offs and I I as a
society we've kind of solved these to
some extent within the past so I'm
hoping that by having the researchers
and the whole community brainstorm and
come up with interesting solutions to
the new things that will happen in the
future
that we can still also push the research
to the Avenue that I think is kind of
the greatest Avenue which is to
understand intelligence right how are we
doing what we're doing and you know
obviously from a scientific standpoint
that is kind of the drive my personal
driver of all the time that I spend
doing what I'm doing really what do you
see the deep learning as a field heading
what do you think the next big big
breakthrough might be so I think deep
learning I I discuss a little of this
before deep learning has to be combined
with some form of discretization program
synthesis I think that's kind of as a
research in itself is an interesting
topic to expand and start doing more
research and then as kind of what will
deep learning and able to do in the
future I don't think that's gonna be
what's gonna happen this year but also
this idea of starting not to throw away
all the weights that the this idea of
learning to learn and really having
these agents not having to restart their
weights and you you can have an agent
that is kind of solving or classifying
images on image net but also generating
speech if you ask it to generate some
speech and and it should really be kind
of almost the same network but might not
be a neural networking might be a neural
network with a optimization algorithm
attached to it but I think this idea of
generalization to new tasks is something
that we first must define good
benchmarks but then I think that's gonna
be exciting and I'm not
sure how close we are but I think
there's the pet if you have a very
limited domain I think we can start
doing some progress and much like how we
did a lot of programs in computer vision
we should start thinking am I really
like a talk that gave that Leon blue to
give gave at ICML a few years ago which
is this train test paradigm should be
broken we we know we should stop
thinking about a training test at
Acharya training set and a test set and
these are closed you know things that
are untouchable I think we should go
beyond these and in meta learning we
call these the meta training set and the
meta test set which is really thinking
about if I know about imagenet why would
that network not work on M NIST which is
a much simpler problem but right now it
really doesn't it you know yeah and but
it just feels wrong right so I think
that's kind of the there's the on the
application or the benchmark sites we we
probably will see quite a few more
interest and progress and hopefully
people defining new and exciting
challenges really do you have any hope
or interest in knowledge graphs within
this context so just kind of totally
yeah constructing graph so going back
that graphs yap well okay neural
networks and graphs but I mean a
different kind of knowledge graph sort
of like semantic graphs or there's
concepts yeah so I think I think the the
idea of graphs is is so I've been quite
interested in sequences first and then
more interesting or different data
structures like graphs and I've studied
graph narrow networks in the last three
years or so I found these models just
very interesting from like deep learning
sites standpoint but then how what do we
want why do we want these models and and
why would we use them what's the
application what's kind of the killer
application of graphs right and perhaps
if we could extract a knowledge
Graff from Wikipedia automatically right
um that would be interesting because
then these graphs have this very
interesting structure that also is a bit
more comfortable with this idea of
programs and deep learning kind of
working together
the jumping neighborhoods and so on you
could imagine defining some primitives
to go around graphs right so I think I
really like the idea of a knowledge
graph and in fact when we we started or
you know as part of the research we did
for StarCraft I thought wouldn't it be
cool to give the graph of you know all
the prerequisites like this all these
buildings that depend on each other and
units that have prerequisites of being
built by that and so this is information
that the network can learn and extract
but it would have been great to see um
or to think of really stack graph as a
giant graph that even also as the game
evolves use kind of star trek taking
branches and so on and we tried we read
a bit of research on these nothing too
relevant but I I really like the idea
and it has elements that are which
something you also worked with in terms
of visualizing your networks as elements
of having human interpretable being able
to generate knowledge representations
that are human interpretable that maybe
human experts can then tweak or at least
understand so there's there's a lot of
interesting aspect there and for me
personally I'm just a huge fan of
Wikipedia and it's it's a shame that our
neural networks aren't taking advantage
of all the structured knowledge that's
on the web what's next for for you
what's next for deep mind what are you
excited about what a four alpha star
yeah so I think the obvious next steps
would be to apply alpha star to other
races I mean that's sort of shows that
the algorithm works because we wouldn't
want to have created by mistake
something in the architecture that
happens to work for Protoss but not for
other races right so as verification I
think that's an obvious next step that
we are working on and then I would like
to see so agents and players can
specialize on
different skill sets that allow them to
be very good I think we've seen alpha
star understanding very well when to
take battles and when to not to do that
do that also very good at
micromanagement and moving the units
around and so on and also very good at
producing non-stop and trading of
economy with building units but I have
not perhaps seen as much as I would like
this idea of the poker idea that you
mentioned right I'm not sure Starcraft
or alpha star rather has developed a
very deep understanding of what the
opponent is doing and reacting to that
and sort of trying to to to trick the
player to do something else or that you
know so this kind of reasoning I would
like to see more so I think purely from
a research standpoint there's perhaps
also quite a few of you things to be
done there in the domain of StarCraft
yeah in a domain of games I've seen some
interesting work in sort of in even
auctions manipulating other players so
forming a belief state and just messing
with people yeah about theory of mind
yeah yeah yeah this is a theory of mine
on star Kirby's kind of they're really
made for each other
yeah so that would be very exciting to
see those techniques applied to
Starcraft or perhaps Starcraft driving
new techniques right as I said this is
always the tension between the two well
Oriol thank you so much for talking
today awesome it was great to be here
thanks
you