Kind: captions
Language: en
the following is a conversation with
charles isbell and michael littman
charles is the dean of the college of
competing at georgia tech
and michael is a computer science
professor at brown university
i've spoken with each of them
individually on this podcast
and since they are good friends in real
life we all thought it would be fun
to have a conversation together quick
mention of each sponsor
followed by some thoughts related to the
episode thank you to
athletic greens the all-in-one drink
that i start
every day with to cover all my
nutritional bases
ate sleep a mattress that cools itself
and gives me yet another reason to enjoy
sleep master class online courses from
some of the most
amazing humans in history and cash app
the app i use to send money
to friends please check out the sponsors
in the description to get a discount
and to support this podcast as a side
note let me say that
having two guests on the podcast is an
experiment that i've been meaning to do
for a while
in particular because down the road i
would like to occasionally
be a kind of moderator for debates
between
people that may disagree in some
interesting ways if you have suggestions
for who you would like to see debate
on this podcast let me know as with all
experiments of this kind it is a
learning process
both the video and the audio might need
improvement
i realized i think i should probably do
three or more cameras next time as
opposed to just two
and also try different ways to mount the
microphone for the
third person also after recording this
intro
i'm going to have to go figure out the
thumbnail
for the video version of the podcast
since i usually put the guest's
head on the thumbnail and now there's
two heads
and two names to try to fit into the
thumbnail it's a kind of bin packing
problem
which in uh theoretical computer science
happens to be an np hard problem
whatever i come up with if you have
better ideas for the thumbnail let me
know as well
and in general i always welcome ideas
how this thing can be improved
if you enjoy it subscribe on youtube
review it with five stars on apple
podcast
follow on spotify support on patreon or
connect with me on twitter
and lex friedman and now here's my
conversation
with charles isbell and michael littman
you'll probably disagree about this
question
but what is your biggest would you say
disagreement
about either something profound and very
important or something completely not
important at all
i don't think we have any disagreements
at all ah i'm not sure that's true
we walked into that one didn't we yeah
so one thing that you sometimes mention
is that and we did this one on air too
as it were
whether or not machine learning is
computational statistics
it's not but it is well it's not
and in particular and more importantly
it is not just computational statistics
so what's missing in the picture what
all the rest of it
what's missing that which is missing oh
because well you can't be wrong now well
it's not just the statistics he doesn't
even believe this
we've had this conversation before if it
were just the statistics then
we would be happy with where we are but
it's not just the statistics that's why
it's computational statistics or if it
were just the computation i agree that
machine learning is not just statistics
it is not just this we can agree on that
nor is it just computational statistics
it's computational statistics it is
computational what is the computational
and computational statistics
does this take us into the realm of
computing it does but i think
perhaps the way i can get him to admit
that uh he's wrong
is that it's about rules it's about
rules
it's about symbols it's about all these
other things statistics it's not about
rules i'm going to say statistics is
about rules but it's not just the
statistics right it's not just a random
variable that you choose and you have a
probability i think you have a narrow
view of statistics
okay well then what would be the broad
view of statistics that would still
allow it to be statistics and not say
history that would make computational
statistics okay
well okay so i i had my first sort of
research mentor
a guy named tom landauer taught me to do
some statistics
right sure and and i was annoyed all the
time because the statistics would say
that what i was doing was not
statistically significant
and i was like but but but and basically
what he said to me is
statistics is how you're going to keep
from lying to yourself
which i thought was really deep it is a
way
to keep yourself honest in a particular
way i agree with that
yeah and so you're trying to find rules
i'm just kind of bringing back to rules
wait wait wait could you possibly try to
define
rules even regular statisticians
non-computational statisticians
do spend some of their time evaluating
rules right applying statistics to try
to understand is this you know is this
does this rule capture this does this
not capture
i mean like hypothesis testing kind of
or like confidence intervals like
like have like more like hypothesis like
i feel like the word statistic literally
means like a summary
like a number that summarizes other
numbers right but i think the field of
statistics actually applies that idea
to things like rules to understand
whether or not a rule is
valid the software engineering
statistics
no programming languages statistics no
because i think there's a very it's
useful
to think about a lot of what ai and
machine learning is or certainly should
be
as software engineering uh as
programming languages
just if to put it in language that you
might understand
in the hyper parameters beyond the
problem the hyper parameters has too
many syllables for me to understand
the hyperparameters of uh that's better
that goes around it right it's the
decisions you choose to make it's the
metrics you choose to use it's the loss
you want to say the practice
of machine learning is different than
the practice of statistics like the
things you have to worry about and how
you worry about them are different
therefore they're different right at a
very little i mean at the very least
it's that that much is true it doesn't
mean that statistics computational or
otherwise aren't important
i think they are i mean i do a lot of
that for example
but i think it goes beyond and i think
that we could think about game theory in
terms of statistics but i don't think
it's very
as useful to do i mean the way i would
think about it or a way i would think
about it
is this way chemistry is just
physics but i don't think it's as useful
to think about chemistry as being just
physics it's useful to think about it as
chemistry the level of abstraction
really matters here so i think it is
there are contexts in which it is useful
that way right so finding that
connection is actually helpful and i
think that's when i when i emphasize the
computational statistics thing i think
i think i want to befriend statistics
and not absorb them here's the here's
the a way to think about it beyond what
i just said right so
what would you say and i want you to
think back to a conversation we had a
very long time ago what would you say is
the difference between
say the early 2000's icml and what we
used to call nips nerfs
was there a difference a lot of the
particularly on the machine learning
that was done there
icmo was around that long oh yeah so
iclear is the new conference
newish uh yeah i guess so and i see him
i was
around the 2000 oh i see male predates
that
i well i think my most cited icml paper
is from 94. yeah
michael knows this better than me
because of course he's significantly
older than i am but the point is
yeah what is the difference what was the
difference between icml and nureps
in the late 90s early 2000s i don't know
what everyone else's perspective would
be but i had a particular perspective at
that time
which is i felt like icml was more of a
of a computer science
place and that nips nerfs was more of an
engineering place
like the kind of math that happened at
the two places
as a computer scientist i felt more
comfortable with the icml math
and the nurbs people would say that
that's because i'm dumb
and that's such an engineering thing to
say so i agree with that part of it but
i
do a little differently actually i had a
nice conversation with tom dietrich
about this
in public on twitter just a couple days
ago i put it a little differently which
is that icml was
machine learning done by uh computer
scientists
and uh nurbs was machine learning done
by computer scientists
trying to impress statisticians
which was weird because it's the same
people at least by the time i started
paying attention
but it just felt very very different and
i think that that perspective of whether
you're trying to impress the
statisticians or you're trying to
impress the programmers is actually very
different and has real impact on
what yeah what you choose to worry about
and what kind of uh
outcomes you come to so i think it
really matters in computational
statistics is a means to an end it is
not an end in some sense
um and i think that really matters here
in the same way that i don't think
computer science is just engineering or
just science or just math or whatever
but
okay so i'd have to now agree that now
we agree on everything
yes yes the important thing here is that
you know my opinions may have changed
but not the fact that i'm right
i think is what what we just came to
right now my opinions may have changed
and not the fact that i'm wrong
that's right i lost me i'm not i think i
lost myself there too but anyway
this happens to us sometimes we're sorry
how does neural networks change
this just to even linger on this topic
change this idea of statistics
how big of a pi statistics is within the
machine learning thing
like because it sounds like hyper
parameters and also just the role of
data
you know this people are starting to use
this terminology software 2.0
which is like the act of programming as
a as a
like you're a designer in the
hyperparameter space of neural networks
and you're also the collector and the
organizer and the cleaner
of the data and that's part of the
programming
uh so how did on the
versus icml topic what's the role of
neural networks and redefining
the size and the role of machine
learning i can't i can't wait to
hear what michael thinks about this but
um i would add one well but
that's true i'll force myself to i think
the the
there's one thing i would add to your
description which is the kind of
software engineering part is what does
it mean to debug for example
but this is a difference between uh the
kind of computational statistics view
of machine learning and the
computational view of machine learning
which is i think one is worried about
the equation as it were and
by the way this is not a value judgment
i just think it's about
perspective but the kind of questions
you would ask when you start asking
yourself what does it mean to program
and develop and build the system
it's a very computer sciencey view of
the problem
i mean when if you get on data science
twitter and econ twitter
you actually hear this a lot with the
uh you know the economist and the data
scientist complaining about the machine
learning people well
you know it's just statistics and i
don't know why they don't don't see this
but they're not even asking the same
questions they're not
thinking about it as a kind of
programming problem
and i think that that really matters
just asking this question i actually
think it's a little different from
programming and hyper parameter space
and sort of collecting
the data but i do think that that
immersion really matters so i'll give
you a quick
a quick example the way i think about
this so i teach machine learning michael
and i have
co-taught a machine learning class which
has now reached i don't know 10 000
people at least over the last several
years or somewhere there's abouts and
my machine learning assignments are of
this form so the super
the first one is something like
implement these five algorithms you know
k
n and s you know svms and boosting and
decision trees and
neural networks and maybe that's it i
can't remember and when i say implement
i mean steal the code
i am completely uninterested you get
zero points for getting the thing to
work
i don't want you spending your time
worrying about uh getting the corner
case right
of you know what happens when you are
trying to normalize distances and the
points on the thing and so you divide by
zero i'm not interested in that right
steal the code however you're going to
run those algorithms
on two data sets the data sets have to
be interesting
what does it mean to be interesting well
data says interesting if it reveals
differences between algorithms which
presumably are all the same
because they can represent whatever they
can represent and two data sets are
interesting together if they
show different differences as it were
and you have to analyze them you have to
justify their interestingness and you
have to analyze in a whole bunch of ways
but all i care about is the data in your
analysis not the programming and i
occasionally end up in these long
discussions with students well
i don't really i copy and paste the
things that i've said the other 15
000 times it's come up which is they go
but the only way to learn
really understand is to code them up
which is a very
programmer software engineering view of
the world if you don't program it you
don't understand it
which is by the way i think is wrong in
a very specific way
but it is a way that you come to
understand because then you have to
wrestle with the algorithm
but the thing about machine learning is
it's not just sorting numbers where in
some sense the data doesn't matter what
matters is
well does algorithm work on these
abstract things and one less than the
other in machine learning
the data matters it does it matters more
than almost anything
and not everything but almost anything
and so
as a result you have to live with the
data and don't get distracted by
the algorithm per se and i think that
that focus on the data
and what it can tell you and what
question it's actually answering for you
as opposed to the question you thought
you were asking is a key and important
thing
about machine learning and is a way that
computationalists as opposed to
statisticians bring a particular
view about how to think about the
process the statisticians by contrast
bring
i i think i'd be willing to say a better
view about the kind of formal
math that's behind it and what an actual
number
ultimately is saying about the data and
those are both important but they're
also different
i didn't really think of it this way is
to build intuition about
the role of data the different
characteristics of data by having two
data sets that are different
and they reveal the differences in the
differences that's that's a really
fascinating that's a really interesting
educational
approach the students love it but not
right away
no they love it later i love it at the
end not at the beginning
not even not even immediately after i
feel like there's a deep
profound lesson about education there
yeah
that uh you can't listen to students
about
whether what you're doing is the right
or the wrong thing
well as a wise uh michael litman once
said to me
about children which i think applies to
teaching is you have to give them
what they need without bending to their
will
and students are like that you have to
figure out what they need you're a
curator your whole job is to curate
and to present because on their own
they're not going to necessarily know
where to search so you're providing
pushes in some direction and learn space
and you have to give them what they need
in a way that keeps them engaged enough
so that they eventually discover
what they want and they get the tools
they need to go and learn other things
what's your view let me put on my
russian hat
which believes that life is like russian
hats by the way if you have one i would
like
those are ridiculous yes but in a
delightful way but sure
what do you think is the role of uh we
talked about balance a little bit
what do you think is the role of
hardship in education
like i think the biggest things i've
learned
like what made me fall in love with math
for example
is by being bad at it until i got good
at it
so like like struggling with a problem
which increased the level of joy i felt
when i finally figured it out
and it always felt with me with teachers
especially modern discussions of
education how can we make
education more fun more engaging more
all those things
or from my perspective it's like you're
maybe missing the point
that education that life is suffering
education is supposed to be hard and
that actually what
increases the joy you feel when you
actually learn something
is that ridiculous do you like to see
your students suffer
okay so this may be a point where we
differ i'd suspect not
i'm gonna do go on well what would your
answer be i wanna hear you first
okay well i would i was gonna not answer
the question
do you know what this dude is i wasn't
gonna hear them suffering no no no no no
i was i was gonna say that there's
i think there's a distinction that you
can make in the kind of suffering right
so
i think you can be in a mode where
you're you're suffering
in a hopeless way versus you're
suffering in a hopeful way
right where you're like you can see
that if you that you still have you can
still imagine getting
to the end right and as long as people
are in that mindset where they're
struggling but
it's not a hopeless kind of struggling
that's
that's productive i think that's really
helpful but it's struggling like if you
break their will
if you leave them hopeless no that don't
sure some people are gonna whatever lift
themselves up by their bootstraps but
like
mostly you give up and certainly it
takes the joy out of it and
you're not going to spend a lot of time
on something that brings you no joy
so it's it's it is a bit of a delicate
balance right you have to thwart people
in a way that they still believe that
there's a way through
right so that's a that we strongly agree
actually so i think
well first off struggling and suffering
aren't the same thing right
being poetic oh no no i actually
appreciate the poetry and
i one of the reasons i appreciate it is
that they are often the same thing and
often quite different right so
you can struggle without suffering you
can certainly suffer and suffer
suffer pretty easily you don't
necessarily have to struggle to suffer
so i think that
you want people to struggle but that
hope matters
you have to they have to understand that
they're going to get through it on the
other side
and it's very easy to confuse the two
i actually think brown university has a
very just
philosophically has a very different
take on the relationship with their
students particularly undergrads from
say
a place like georgia tech which is which
universities better
uh well i have my opinions on that i
mean remember charles said
it doesn't matter what the facts are i'm
always right the correct thing
is that it doesn't matter they're
different um but
clearly he went to a school
like the school where he is as an
undergrad i went to a school
specifically the same school though it
was it changed a bit in the in the
intervening years
brown or georgia tech no i was talking
about georgia tech and i went yeah
and i went to an undergrad place that's
a lot like the place where i work now
and so it does seem
like we're more familiar with these
models there's a similarity between
brown and yellow
yeah there's a i think that i think
they're quite similar yeah and duke
duke has some similarities too but it's
got a little southern
draw you've kind of worked here you sort
of worked at universities that are like
the places where
you learned and
the same would be true for me are you
uncomfortable uh
venturing outside the box is that what
you're saying
journeying out what i'm saying yeah
charles is definitely he only goes to
places that have institute in the name
right it has worked out that way well
academic places anyway
well no i was a visiting scientist at
upenn or visiting
visiting something at upenn oh wow i
just i just understood
your joke which one
five minutes later i like to set these
sort of time bombs
the institute is in the uh uh that
charles only goes to places that have
institute
in the name so i guess georgia
i forget that georgia tech is georgia
institute of technology the number of
people who refer to it as georgia tech
university is large and incredibly
irritating
that's one of the few things that
generally gets under my schedule but
like schools like georgia tech and mit
have as part of the ethos like there is
i want to say there's a there's an
abbreviation
that someone taught me like i htfp
something like that like there's a
there's a
there's an expression which is basically
i hate being here which they say
so proudly and that is definitely not
the ethos at brown like brown is
there's a little more pampering and
empowerment and stuff and it's not like
we're gonna crush you and you're gonna
love it
so yeah i think there's a i think the
ethos
are different mm-hmm that's interesting
yeah we had drone proofing
what's that trump graduate from georgia
tech this is a true thing feel free to
look it up
uh if you a lot of schools have this by
the way
no actually georgette was barely the
first brandeis has it had it
i feel like georgia tech was the first
in the look first of all
it was it was the first time i think um
had the first time
stop that first masters in computer
science actually right online masters
well that too but way back in the 60s um
nsf yeah yeah
you're the first information and
computer science master's degree in the
country
um but the uh georgia tech it used to be
the case in order to graduate from
georgia tech uh you had to take a drown
proofing class
where effectively they threw you water
tied you up
if you didn't drown you got to graduate
i believe so
there were certainly versions of it but
i mean luckily they ended it just before
i had to graduate because otherwise
would have never graduated
it wasn't going to happen uh i want to
say 84 or 83 someone around then they
they ended it but uh yeah you used to
have to prove you could tread water for
some ridiculous amount of time are you
two yeah you couldn't graduate no it was
more than two hours two minutes
okay it was in a bathtub
it was in a pool but it was a real thing
but that idea that you know push you
fully clothed yeah fully clothed i don't
think i bet it was that and not tied up
because like who needs to learn how to
swim when you're tied
nobody but who needs to learn when to
swim when you're actually falling into
the water dressed that's a real thing
i think your facts are getting in the
way with a good story oh that's fair
that's fair i didn't
think all right so they didn't tell you
what the narrative mattered but whatever
it was you had to it was called drown
proofing for a reason the point of the
story
michael uh is struggle it it's well no
but that's good it doesn't
bring it back to struggle that's a part
of what georgia tech has always been and
we struggle with that by the way
uh about what we want to be as things go
but you you sort of how much can you
be pushed without breaking and you come
out of the other end stronger right
there
there's this saying we said when i was
an undergrad there which is georgia tech
building tomorrow the night before
right kind of idea that you know
give me something impossible to do and
i'll do it in a couple of days because
that's what i just spent the last four
or five
or six years that ethos definitely stuck
to you
having now done a number of projects
with you you definitely will do it the
night before that's not entirely true
there's nothing wrong with waiting until
the last minute the secret is knowing
when the last minute is
right that's brilliant that's
brilliantly put yeah that yeah that's
that is a definite
charles statement that i am trying not
to embrace
and i appreciate that because you helped
move my last minute that's the social
construct that we converge together what
the definition of last minute is and we
we figure that out all together in fact
mit
you know i'm sure a lot of universities
have this but mit has like mit time that
yeah everyone has always agreed together
that
there is such a concept and everyone
just keeps showing up like 10 to 15
to 20 depending on the department late
to everything
so there's like a weird drift that
happens it's kind of fascinating yeah
we're five minutes
five minutes in fact the classes will
say you know well this is no longer true
actually
but it used to be a class was started
eight but actually started 805
yeah it ends at nine actually ends at 8
55. uh everything's five minutes off and
nobody expects anything to start until
five minutes after the half hour or
whatever it is
it still exists it hurts my head well
let's rewind the clock
back to the 50s and 60s when you guys
met
how did you i'm just kidding i don't
know but what can you tell the story of
how you met so
you've like the internet and the world
kind of knows you as
as as connected in some ways
in terms of education of teaching the
world that's
that's like the public facing thing but
how did you as human beings
and as collaborators
meet i think there's two stories one is
how we met
and the other is how we got to know each
other
i'm not gonna say fellaini i'm gonna say
that we came to understand that we
had some common something yeah it's
funny because on the surface i think
we're
we're different in a lot of ways but
there's something yeah i mean that's
just consonant there you go afternoon
so i will tell the story of how we met
and i'll let michael tell the story of
how we
okay all right okay so here's how we met
um i was already at that point it was
18t labs
there's a long interesting story there
but anyway i was there and uh
michael was coming to interview he was a
professor at duke at the time but
decided for reasons that he wanted to be
in new jersey
uh and so that would mean uh bell lab
slash att labs
uh and we were doing interview
interviews very much like academic
interviews uh and so i had to be there
uh we all had to meet with him
afterwards and so on one on one
but it was obvious to me that he was
gonna be hired
like no matter what because everyone
loved him they were just talking about
all the great stuff he did and
oh he did this great thing and you just
won something at triple a i think or
maybe you got 18 papers in triple either
but
i got the best paper award at your play
for the crosswords right exactly
so that it all happened and everyone was
going on and on and on about actually
tinder was saying incredibly nice things
about you really yes so he can be very
grumpy yes that's very that's nice to
hear he was grumpily saying very nice
things oh that's that makes sense and
that does make sense so you know so
it was going to come so why were we why
was i meeting him i had something else i
had to do i came here what it was yeah
it probably involved commenting he
remembers meeting me as inconveniencing
his afternoon
so he came so eventually came to my
office i was in the middle trying to do
something i can't remember what and he
came and he sat down and for
reasons that are purely accidental
despite what michael thinks
my desk at the time was set up in such a
way that had
sort of an l shape and the chair on the
outside was always lower than the chair
that i was in
and you know the kind of point was the
only reason i think that was on purpose
is because you told me it was on purpose
i don't remember that anyway the thing
is that you know it kind of his guest
chair was really low so that he could
yeah he could look down at everybody the
idea was just to simply create a nice
environment that you were asking for a
mortgage and i was going to say no that
was a
very simple idea here anyway so we sat
there and we just talked for a little
while and i think he got the impression
that i didn't like him
that wasn't true strongly the talk was
really good
by the way it was terrible and after
right after the talk i said to my host
michael kearns who ultimately was
my boss i'm a huge fan i'm a friend and
a huge fan of michael yeah yeah he is a
remarkable
person um i i after my talk today
i went into this i went back at ball
he's good at that
basketball no but basketball racquetball
squash which is not
racquetball yes squash no and i hope you
you hear that michael
you mean like your parents as a game not
his skill level because i'm pretty sure
he's all right there's some
competitiveness there but the point
is that it was like the middle of the
day i had full day of interviews like i
met with people but then in the middle
of the day i gave
a job talk and then um and then there
was going to be more interviews but
i i pulled michael aside and i said
i think it's in both of our best
interests if i just leave now
because that was so bad that it's just
be embarrassing if i have to talk to any
more people like you look bad for having
invited me
like it's just let's just forget this
ever happened
so i don't think the talk went well it's
one of the most michael littman set of
sentences i think i've ever heard
he did great or at least everyone knew
he was great so maybe it didn't matter
i was there i remember the talk and i
remember him being very much the way i
remember him now
in any given week so it was good and we
met and we talked about stuff he thinks
i didn't like him but because he was so
grumpy
must been the chair thing the chair
thing and the low voice i think
the like obviously and that like that
like slight like
skeptical look yeah i have no idea what
you're talking about
well i probably didn't have any idea
what you were talking about
anyway i liked him he asked me questions
i answered questions i felt bad about
myself it was a normal day
then he left and then he left and that's
how you tell me can we take it
and then i got hired and i was in the
group can we take a slight tangent on
that on this topic of
it sounds like uh maybe you could speak
to the bigger picture
it sounds like you're quite
self-critical who charles
no you oh i think i can i can do better
i can do better i'll
try me again i'll i'll do better
yeah that was like a like a three out of
ten responses
so let's try to work it up to five and
six uh you know i remember
uh marvin minsky said uh on on a video
interview
something that the key to success in
academic research is to hate everything
you do
for some reason i think i followed that
because i hate everything he's done
[Laughter]
uh it's a good line that's a success
maybe that's a keeper but um but do you
do find that resonates with you at all
in how you think about talks and so on
i would say it differently it's not
really that's such an mit view of the
world though
so i remember i i remember talking about
this when uh as a student you know
you were basically told uh i will clean
it up for the purpose of the podcast
um uh my work is crap my work is crap my
work is crap my work is crap then you
like go to a conference or something
like everybody else's work is crap
everybody else is working crap and you
feel better and better about it yeah
uh relatively speaking and then you sort
of keep working on it
i don't hate my work that resonates with
me yes i've never hated my work but i
have
i have been dissatisfied with it
and i think being dissatisfied being
okay with the fact that you've taken a
positive step the derivative is positive
maybe even the second derivative is
positive that's important because that's
a part of the the hope right
but you have to but i haven't gotten
there yet if that's not there that i
haven't gotten there yet
then you know it's hard to it's hard to
move forward i think so i buy that
which is a little different from hating
everything that you do yeah i mean
there's
there's things that i've done that i
like better than i like myself
so it's separating me from the work
essentially so i think i am very
critical of myself
but sometimes the work i'm really
excited about and sometimes i think it
doesn't happen right away so i found the
work that i've
liked that i've done most of it
i liked it in retrospect more when i was
far away from it in time
i have to be fairly excited about it to
get done
no excited at the time but then happy
with the result or but years later or
even i might go back you know what
that actually turned out to be yeah that
turned out to matter or oh gosh it turns
out i've been thinking about that
it's actually influenced all the work
that i've done since without realizing
it
but that guy was smart yeah that guy had
a future
yeah yeah he's going places
i think there's so yeah so i think
there's something to it i think there's
something to the idea you've got to
you know hate what you do but it's not
quite hate it's just being unsatisfied
and different people motivate themselves
differently i don't happen to motivate
myself with self-loathing
i happen to motivate myself so you're
able to sit back and
be proud of in retrospect of the work
you've done
well and it's easier when you can
connect with other people because then
you can be proud of them
a lot of the people yeah and then the
questions you can still safely hate
yourself
it's a win-win michael or at least win
lose which is what you're looking for
oh wow there's so many brilliant lines
in this
there's levels uh so how did you
actually meet me
yeah so my the way i think about it is
because we didn't do much
research together at 18t but um but then
we all got laid off
so so that was that by the way i decided
to interrupt but that was like
one of the most magical places
historically speaking
they did not appreciate what they had
and how do we uh i feel like there's a
profound lesson in there too uh
how do we get it like what was why was
it so magical is just the coincidence of
history
or is there something special some
really good managers and people who
really believed in
machine learning as this is going to be
important
um let's get the the people who are
thinking about this in creative and
and insightful ways and put them in one
place and
stir yeah but even beyond that right it
was
it was bell labs at its heyday and even
when we were there which i think was
past it
to be clear he's gotten to be at bell
labs i never got to be at bell labs
i joined after that yeah i should have
been 91 as a grad student
so i was there for a long time um every
summer except
twice i worked for companies that had
just stopped being better labs right
bell core and then att labs so about
labs was
several locations or for the for the
research or is it what like
jerseys are involved somehow they're all
in jersey yeah they're all over the
place but they're in a couple places
murray hill was the bell labs um
so you you had you had an office in mary
hill at one point in your career
yeah and i i played ultimate frisbee on
the cricket pitch at bell labs at murray
hill
uh and then it became 18t labs when
split off with loose during what we
called uh tri-vestiture supposedly
better than michael koren's ultimate
frisbee yeah oh yeah
okay but i think that one's not boasting
i think that i think charles plays a lot
of ultimate and i don't think mike i was
yes but but that wasn't the point the
point is yes yes
sorry okay i have played on a
championship winning ultimate frisbee
team
or whatever ultimate team with charles
so i know
how good he is he's really good how good
i was anyway when i was younger but the
thing is i know how young he was when he
was yeah that's true
that was true so much younger than now
he's old enough yeah i'm older michael
is a much
was a much better basketball player than
i was michael kearns yes no not michael
i'm very clear so you don't know how
terrible i am
but you have a probably pretty good
guess that you're not as good as michael
kearns
he's tall and and he cared about it very
outlet he's very good he's
probably competitive i love hanging out
with michael anyway but we were talking
about something else although i no
longer remember what it was what were we
talking about
but also labs so so uh this was kind of
cool about what was magical about it
the first thing you have to know is that
bell labs was an arm of the government
right because att was an army of
government
it was a monopoly uh and you know every
month you paid a little
thing on your phone bill which turned
out was a tax for like all the research
that bell labs was doing
and you know they invented transistors
and the laser and whatever else is that
big bang or whatever the
cosmic background radiation yeah they
did all that stuff they had some amazing
stuff with directional microphones by
the way i got to go in this room
um where they they had all these panels
and everything
and we would talk and one another and he
moved some panels around and then
he would have me step two steps to the
left and i couldn't hear a thing he was
saying because nothing was bouncing off
the walls
and then he would shut it all down and
you could hear your heartbeat yeah
deeply disturbing to hear your heart
beat you can feel it i mean you can feel
it now there's so much all this sort of
noise around anyway bill labs is about
pure research
it was a university in some sense the
purest sense of a university
but without students so it was all the
faculty working with one another
and students would come in to learn they
would come in for three or four months
you know during the summer and they
would go away
but it was just this kind of wonderful
experience i could walk out my door
in fact i would often have to walk out
my door and deal with rich sutton and
michael kearns yelling at each other
about whatever it is they were yelling
about
the proper way to prove something or
another and i could just do that and
dave mcallister and evan
and peter stone and and all of these
other people including
satinder and then eventually michael and
it was just a place where you could
think
thoughts and it was okay because so long
as once every 25 years or so
somebody invented a transistor it paid
for everything else you could afford to
take the risk
and then when that all went away it
became harder
and harder and harder to justify it as
far as the folks who were very far away
were concerned
and there was such a fast turnaround
among middle management
on the atnt side that you never had a
chance to really build the relationship
at least people like us didn't have a
chance to
to build relationships so when the
diaspora happened um
it was amazing right yeah everybody left
and i think everybody ended up at a
great place and
made a huge made a continued to do
really good work with with machine
learning but it was a wonderful place
and people will ask me you know what's
the best job you you've ever had
and as a professor anyway the answer
that i would give is
um well probably
bell labs in some very real sense and i
would never have a job like that again
because bell labs doesn't exist anymore
and you know microsoft research is great
and google does good stuff and you can
pick ibm
you can tell if you want to but bell
labs was magical it was around for it
was an important time
and it represents a a high water mark in
in basic research in the u.s is there
something you could say about the
physical proximity and the chance
collisions
like we live in this time of the
pandemic where everyone
is maybe trying to see the silver lining
and accepting the remote nature of
things
is is there one of the things that
people like faculty
that i talk to miss is the
the procrastination like the chance to
like everything is about meetings that
are supposed to be there's not a chance
to just
uh you know talk about comic book or
whatever like go into discussion that's
totally pointless
so it's funny you say this because
that's how we met matt
it's exactly that so i'll let michael
say that but i'll just add one thing
which is just that uh
you know research is a social process
and it helps to have
random social interactions even if they
don't feel social at the time that's how
you get things done
one of the great things about the a lab
when i was there i
don't quite know what it looks like now
once they moved buildings but we had
entire walls that were whiteboards and
people would just get up there and they
were just right
and people would walk up and you'd have
arguments and you'd explain things to
one another
and you got so much out of the freedom
to do that you had to be
okay with people challenging every
freaking word you said which i would
sometimes find
deeply irritating but most of the time
it was it was quite useful but the sort
of pointlessness and the interaction was
in some sense the point at least for me
yeah i mean you
i think offline yesterday i mentioned
josh tannenbaum and he's
very much he put he's a man he's such an
inspiration
in in the child like
way that he pulls you in on any topic it
doesn't even have to be about machine
learning
it could or or the brain he'll just pull
you into a closest
writable surface which is uh still you
can find whiteboards at mit everywhere
and and just like uh like basically
cancel all meetings and talk for a
couple hours about some
some aimless thing and it it feels like
the whole world the time
space continuum kind of warps and that
becomes the most important thing
and then it's just it's so true it's
it's
definitely something worth missing in
this in this world where everything's
remote
there's some magic to the physical
presence whenever i wonder myself
whether mit really is as great as i
remember it i just go talk to josh yeah
you know that's funny is
there's a few people in this world that
carry the
the best of what particular institutions
stand for right and it's uh it's josh
i mean i i don't i my guess is he's
unaware of this
that's the point that the masters are
not
aware of their mastery so how do we all
meet
yes but but first a tangent no
how did you meet me so i'm not sure what
you were thinking of but my
when it started to dawn on me that maybe
we had a longer-term bond
was after we all got laid off and
you had decided at that point that there
we were still paid we were given an
opportunity to like do job search and
kind of make a transition
but it was clear that we were done and
i would go to my office to work and you
would go to my office to keep me from
working
that was that was my recollection of it
and you had decided that there was no
really no point in working for the
company because the company our
relationship with the company was
was done yeah but remember i felt that
way beforehand it wasn't about the
company it was about the set of people
there doing really cool things and it
always
always been that way but we were working
on something together oh yeah
yeah that's right oh so at the very end
we all got laid off but then
our boss came to our boss's boss came to
us
because our boss was michael kearns and
he had jumped ship
brilliantly like perfect timing like
things like right before the ship was
about to sink
he was like gotta go and and and
landed perfectly because michael kearns
because michael king
and um leaving the rest of us to go like
this is fine and then it was clear that
wasn't fine and we were all
toast so we had this sort of long period
of time but then our boss figured out
okay wait maybe we can save a couple of
these people
if we can have them do something really
useful
and uh the useful thing was we were
going to make a
basically an automated assistant that
could help you with your calendar you
could like
tell it things and it would it would
respond appropriately it would just kind
of integrate across
all sorts of your personal information
and so me and charles and peter stone
were this were set up as the crack team
to actually solve this problem
uh other people maybe were too
theoretical that they thought and
and but we could actually get something
done so we sat down to get something
done and there wasn't time
and it wouldn't have saved us anyway and
so it all kind of went downhill
but the interesting i think coda to that
is that our boss's boss is a guy named
ron brockman
and he when he left at t
because we were all laid off he went to
darpa
started up a program there that became
kalo
which is the program from which siri
sprung
which is a digital assistant that helps
you with your calendar and a bunch of
other things
um it really you know in some ways got
its start
with me and charles and peter trying to
implement this vision that ron brockman
had that
he ultimately got implemented through
his role at darpa
so when i'm trying to feel less bad
about having been laid off from
what is possibly the greatest job of all
time
i think about well we kind of helped
birth siri
that's something and he did other things
too but
the we got to spend a lot of time in his
office and
talk about we got to spend a lot of time
in my office yeah
yeah yeah and so uh so then we went on
our merry way
everyone went to different places
charles landed at georgia tech which was
uh what he always dreamed he would do
and so
um that worked out well yeah um i
came up with a saying at the time which
is luck favors the charles
it's kind of like luck favors the
prepared but charles like
like he'd wish something and then it
would basically happen just the way he
wanted it was
it was inspirational to see things go
that way things worked out and we stayed
in touch
and then um i think it really helped
when you were working on i mean you kept
me in the loop for things like threads
and the work that you were doing at
georgia tech but then
when they were starting their online
master's program he knew that i was
really excited about
moocs and online teaching and he's like
i have a plan and i'm like tell me your
plan he's like i can't tell you the plan
yet because they were
deep in in negotiations between georgia
tech and udacity to make this happen
and they didn't want it to leak so
charles would kept teasing me about it
but wouldn't tell me what was actually
going on and eventually it was announced
and he said i would like you to teach
the machine learning course with me
i'm like that can't possibly work um but
it was a great idea and it was
it was super fun it was a lot of work to
put together but it was it was really
great
and was that the first time you thought
about first of all
was it the first time you got seriously
into teaching
i mean you know i'm trying to get the
feeling right i'll tell you this is
already
after you jump to so like there's a
little bit of
jumping around in time yeah sorry about
it there's a pretty big jump in time so
like the moocs
thing so charles got to georgia tech and
he i mean maybe charles maybe this is a
trick
in 2002. he got to georgia tech in 2002
and um but then and worked on things
like revamping the curriculum the
undergraduate curriculum so that it had
some kind of
semblance of modular structure because
computer science was
at the time moving from a fairly narrow
specific set of topics to touching a lot
of other parts
of of of intellectual life and the
curriculum
was supposed to reflect that and so um
charles played a big role in
in kind of redesigning that and then and
for my and for my
my labors i ended up his associate dean
right he got to become an associate dean
of in charge of educational stuff well
this would be a valuable lesson if
you're
good at something uh they will give you
responsibility to do more of that thing
well until you don't show confidence
don't show confidence if you
well you know what the responsibility
here's what they say yeah
the reward for good work is more work
the reward for bad work is less work
which i don't know depending about what
you're trying to do that week
one of those is better than the other
well one of the problems with the word
work sorry to interrupt
is that it's seems to be an antonym
in this particular language we have the
opposite of happiness
but it seems like they're they're like
that's one of you know we talked about
balance it's uh it's always like
work-life balance it always rubbed me
the wrong way as
a terminology i know it's just words
right the opposite of work is play
but yeah ideally work is play oh i can't
tell you how much time i'd spend
certainly i was about labs except for a
few very key moments
uh as a professor i would do this too i
was just saying cannot believe they're
paying me to do that um because it's fun
it's something that i would i would do
for a hobby
if i could anyway uh so that sort of
worked out i'm sure you want to be
saying that
when this is being recorded as a dean
that is not true at all
i need a raise yes but but i think here
with with this that even though a lot of
time passed
you know michael and i talked almost
every well we texted almost every day
during the period charles at one point
took me
there was the icml conference the
machine learning conference was in
atlanta i was the chair the general
chair of the conference
charles was my publicity chair or
something like that or
something fundraising champion sure yeah
um but he decided it'd be really funny
if he didn't actually show up for the
conference in his own home city
uh so he didn't but he did at one point
picked me up at the conference in his
tesla and drove me to the atlanta mall
and forced me to buy an iphone because
he didn't like
how it was to text with me and thought
it would be better for him
if i had an iphone the text would be
somehow smoother
and it was and it was and it is and his
life is better and my life is better and
so
death but but it was yeah charles
forcing me to get an iphone
so that he could text me more
efficiently i thought that was an
interesting moment it works for me
anyway so we kept talking the whole time
and then eventually we did the
we did the teaching thing and it was
great and ther