Kind: captions
Language: en
the following is a conversation with
Daphne Koller a professor of computer
science at Stanford University a
co-founder of Coursera with Andrew Eng
and founder and CEO of in seat row a
company at the intersection of machine
learning and biomedicine we're now in
the exciting early days of using the
data-driven methods of machine learning
to help discover and develop new drugs
and treatments at scale Daphne and in
seat row are leading the way on this
with breakthroughs they may ripple
through all fields of medicine including
ones most critical for helping with a
current coronavirus pandemic this
conversation was recorded before the
cove 8:19 outbreak for everyone feeling
the medical psychological and financial
burden of this crisis
I'm sending love your way stay strong
we're in this together we'll beat this
thing this is the artificial
intelligence podcast if you enjoy it
subscribe I need to review it with five
stars an apple podcast supported on
patreon are simply connected me on
Twitter Alex Friedman spelled Fri D ma M
as usual I'll do a few minutes of ads
now and never any ads in the middle that
can break the flow of this conversation
I hope that works for you and doesn't
hurt the listening experience this show
is presented by cash app the number-one
finance app in the App Store when you
get it used collects podcast cash app
lets you send money to friends buy
Bitcoin and invest in the stock market
with as little as one dollar since
ketchup allows you to send and receive
money digitally peer-to-peer and
security in all digital transactions is
very important and you mentioned that
PCI data security standard the cash shop
is compliant with I'm a big fan of
standards for safety and security PCI
DSS is a good example of that where a
bunch of competitors got together and
agreed that there needs to be a global
standard around the security of
transactions now we just need to do the
same for Thomas vehicles and the ad
systems in general so again if you get
cash app from the App Store Google Play
and use the code luxe podcast you get
ten dollars in cash that will also
donate ten dollars to first an
organization that is helping to advance
robotics
and STEM education for young people
around the world and now here's my
conversation with Daphne Koller
so you co-founded Coursera I made a huge
impact in the global education of AI and
after five years in August 2016 wrote a
blog post saying that you're stepping
away and wrote quote it's time for me to
turn to another critical challenge the
development of machine learning and it's
applications to improving human health
so let me ask two far-out philosophical
questions one do you think will one day
find cures for all major diseases known
today and two do you think will one day
figure out a way to extend the human
lifespan perhaps to the point of
immortality so one day is a very long
time and I don't like to make
predictions of the type we will never be
able to do X because I think that's a
you know that's the smacks of hubris it
seems that never and in in in the entire
eternity of human existence will we be
able to solve a problem that being said
curing disease is very hard because
oftentimes by the time you discover the
disease a lot of damage has already been
done and so to assume that we would be
able to cure disease at that stage
assumes that we would come up with ways
is basically regenerating entire parts
of the human body in the way that
actually returns it to its original
state and that's a very challenging
problem we have cured very few diseases
we've been able to provide treatment for
an increasingly large number but the
number of things that you could actually
define to be cures is actually not that
large so I think that's it there's a lot
of work that would need to happen for
one could legitimately say that we have
cured even a reasonable number of far
less all diseases on the scale of 0 to
100 where are we in understanding the
fundamental mechanisms of all major
diseases what's your sense so from the
computer science perspective that you've
entered the world of health how far
along are we
I think it depends on which disease I
mean there are ones where I would say
we're maybe not quite at a hundred
because biology is really complicated
and there's always new things that we
uncover that people didn't even realize
existed so but I would say there's
diseases where we might be in the
seventies or eighties and then there's
diseases in which I would say probably
the majority where we're really close to
zero with Alzheimer's and schizophrenia
and type 2 diabetes fall closer to zero
or to the 80
I think Alzheimer's is probably closer
to zero than to 80 there are hypotheses
but I don't think those hypotheses have
as of yet been sufficiently validated
that we believe them to be true and
there is an increasing number of people
who believe there's a traditional
hypotheses might not really explain
what's going on I would also say that
Alzheimer's and schizophrenia and in
even type 2 diabetes are not really one
disease they're almost certainly a
heterogeneous collection of mechanisms
that manifests in clinically similar
ways so in the same way that we now
understand that breast cancer is really
not one disease it is multitude of
cellular mechanisms all of which
ultimately translate to uncontrolled
proliferation but it's not one disease
the same is almost undoubtedly true for
those other diseases as well that
understanding that needs to precede any
understanding of the specific mechanisms
of any of those other diseases now in
schizophrenia I would say we're almost
certainly closer to zero than to
anything else type 2 diabetes is a bit
of a mix there are clear mechanisms that
are implicated that I think have been
validated they have to do with insulin
resistance and such but there's almost
certainly there as well many mechanisms
that we have not yet understood you've
also thought and worked a little bit on
the longevity side do you see the
disease and longevity as
overlapping completely partially or not
at all as efforts those mechanisms are
certainly overlapping there's a
well-known phenomenon that says that for
most diseases other than childhood
diseases the risk for getting for
contracting that disease increases
exponentially year-on-year every year
from the time you're about 40 so
obviously there is a connection between
those two things I that's not to say
that they're identical there's clearly
aging that happens that is not really
associated with any specific disease and
there's also diseases and mechanisms of
disease that are not specifically
related to aging so I think overlap is
where we're at okay it is a little
unfortunate that would get older and it
seems that there's some correlation with
the fact the the occurrence of diseases
or the fact that we'll get all there
mm-hmm and both are quite sad I mean
there's processes that happen as cells
age that I think are contributing to
disease some of those have to do with
the DNA damage that accumulates the
cells divide where the repair mechanisms
don't fully it correct for those there
are accumulations of proteins that are
misfolded and potentially aggregate and
those two contributes a disease and
contribute to inflammation there is an
um there's a multitude of mechanisms
that have been uncovered that are sort
of wear and tear at the cellular level
that contribute to disease processes
that and I'm sure there's many that we
don't yet understand on a small tangent
perhaps philosophical this uh the the
fact that things get older and the fact
that things die is a very powerful
feature for the growth of new things
that you know it's a learning it's a
kind of learning mechanism so it's both
tragic and beautiful so do you do you do
you so in you know in trying to fight
disease
and trying to fight aging do you think
about sort of the useful fact of our
mortality or would you like what if you
were could be immortal would you choose
to be immortal
again I think immortal is a very long
time I don't know that that would
necessarily be something that I would
want to aspire to but I think all of us
aspire to an increased health span I
would say which is an increased amount
of time where you're healthy and active
and feel as you did when you were 20 and
we're nowhere close to that people
deteriorate physically and mentally over
time and that is a very sad phenomenon
so I think a wonderful aspiration would
be if we could all live to you know the
biblical 120 may be in perfect health in
my quality of life high quality of life
I think that would be an amazing goal
for us to achieve as a society now is
the right age 120 or 100 or 150 I think
that's up for debate but I think an
increased health span is a really worthy
goal and anyway in a grand time the age
of the universe it's all pretty short so
from the perspective you've done
obviously a lot of incredible work on
machine learning so what role do you
think data and machine learning play in
this and his goal of trying to
understand diseases in trying to
eradicate diseases up until now I don't
think it's played very much of a
significant role because largely the
data sets that one really needed to
enable a powerful machine learning
methods those data sets haven't really
existed there's been dribs and drabs and
some interesting machine learning that
has been applied I would say machine
learning / data science but the last few
years are starting to change thoughts so
we now see an increase in some large
data set
but equally importantly an increase in
technologies that are able to produce
data at scale it's not typically the
case that people have deliberately
proactively used those tools for the
purpose of generating data for machine
learning they to the extent that those
techniques have been used for data
production they've been used for data
production to drive scientific discovery
and the machine learning came as a sort
of by-product second stage of oh you
know now we have a data set let's do
machine learning on that rather than a
more simplistic data analysis method but
what we are doing it in seat rows
actually flipping that around and saying
here's this incredible repertoire of
methods that bile engineers cell
biologists have come up with let's see
if we can put them together in brand-new
ways with the goal of creating data sets
that machine learning can really be
applied on productively to create
powerful predictive models that can help
us address fundamental problems in human
health so really focus to get make data
the the primary focus and the primary
goal and find use the mechanisms of
biology and chemistry to to uh to create
the kinds of data set that could allow a
machine learning to benefit the most I
wouldn't put it in those terms because
that says the data is the end goal
data's the means so for us the end goal
is helping address challenges in human
health and the method that we've elected
to do that is to apply machine learning
to build predictive models and machine
learning in my opinion can only be
really successfully applied especially
the more powerful models if you give it
data that is of sufficient scale and
sufficient quality so how do you create
those data sets so as to drive the
ability to generate predictive models
which subsequently help improve human
health so before we dive into the
details of that even take a step back
and ask when and where was your interest
in human health born are there moments
events perhaps
if I may ask tragedies in your own life
that catalyzes passion or was at the
broader desire to help humankind so I
would say it's a bit of both so on I
mean my interest in human health
actually dates back to the early 2000s
when when a lot of my peers and machine
learning and I were using datasets that
frankly we're not very inspiring some of
us old-timers still remember the
quote-unquote twenty newsgroups dataset
where it was literally a bunch of text
from twenty newsgroups a concept that
doesn't really even exist anymore and
the question was can you classify which
which news group a particular bag of
words came from and it wasn't very
interesting the datasets at the time on
the biology side were much more
interesting both from a technical and
also from an aspirational perspective
they were still pretty small but they
were better than 20 news groups and so I
started out I think just by just by
wanting to do something that was more I
don't know societally useful and
technically interesting and then over
time became more and more interested in
the biology in the and the human health
aspects for themselves and began to work
even sometimes on papers that were just
in biology without having a significant
machine learning component I think my
interest in drug discovery is partly due
to an incident I had with when my father
sadly passed away about 12 years ago he
had an autoimmune disease that settled
in his lungs and the doctors basis it
well there was only one thing we could
do which is give him prednisone at some
point I remember doctor even came and
said hey let's do a lung biopsy to
figure out which autoimmune disease he
has and I said would that be helpful
would that change treatments no there's
only prednisone that's the only thing we
can give him and I have friends who were
rheumatologist who said the FDA would
never approve press his own today
because
the ratio of side effects to benefit is
probably not large enough today we're in
a state where there's probably four or
five maybe even more well depends for
which autoimmune disease but there are
multiple drugs that can help people with
autoimmune disease and many of which can
exist at 12 years ago and I think we're
at a golden time in some ways and drug
discovery where there's the ability to
create drugs that are much more safe for
much more effective than we've ever been
able to before and what's lacking is
enough understanding of biology and
mechanism to know where to aim that
weird ain't that engine and I think
that's where machine learning can help
so in 2018 he started and now lead a
company in seat row which is a like you
mentioned perhaps the focus is drug
discovery and the utilization of machine
learning for drug discovery so you
mentioned that quote we're really
interested in creating what you might
call a disease in a dish model disease
in a dish models places where disease is
a complex where we really haven't had a
good model system or typical animal
models that have been used for years
including testing on mice just aren't
very effective so can you can you try to
describe what is an animal model and
what what is a disease in a dish model
sure so an animal models for disease is
where you create effectively its what it
sounds like it's it's a oftentimes a
mouse where we have introduced some
external perturbation that creates the
disease and then we cure that disease
and the hope is that by doing that we
will cure a similar disease in human the
problem is is that oftentimes the way in
which we generate the disease and the
animal has nothing to do with how that
disease actually comes about in a human
it's what you might think of as a copy
of the of
phenotype a copy of the clinical outcome
but the mechanisms are quite different
and so curing the disease in the animal
which in most cases doesn't happen
naturally mice don't get Alzheimer's
they don't get diabetes they don't get
atherosclerosis they don't get autism or
schizophrenia those cures don't
translate over to what happens in the
human and that's where most drugs fails
just because the findings that we had in
the mouse don't translate to a human the
disease in the dish bottles is a fairly
new approach it's been enabled by
technologies that have not existed for
more than five to ten years so for
instance the ability for us to take a
cell from any one of us you or me revert
thats a skin cell to what's called stem
cell status which is a what if it was
called a pluripotent cell that can then
be differentiated into different types
of cells so from that flurry potent cell
one can create a wax neuron or a lex
cardiomyocyte or alexa parasite that has
your genetics but that right our cell
type and so if there is a genetic burden
of disease that would manifest in that
particular cell type you might be able
to see it by looking at those cells and
saying oh that's what potentially sick
cells look like versus healthy cells and
understand how and then explore what
kind of interventions might revert the
unhealthy looking cell to a healthy cell
now of course curing cells is not the
same as curing people and so there's
still potentially translate ability gap
but at least for diseases that are
driven say by human genetics and where
the human genetics is what drives the
cellular phenotype there is some reason
to hope that if we revert those cells in
which the disease begins and where the
disease is driven by genetics and we can
revert that cell back to a healthy state
maybe that will help also
the more global clinical phenotypes
that's really what we're hoping to do
that step that backward step I was
reading about it the Yamanaka factor yes
so think that the reverse step back to
stem cells yes I think seems like magic
it is I'm honestly before that happened
I think very few people would have
predicted that to be possible it's
amazing can you maybe elaborate is it
actually possible like word like how
state so this result was maybe like I
don't know how many years ago maybe ten
years ago was first demonstrated
something like that
is this how hard is this like how noisy
is this backward step it seems quite
incredible and cool it is it is
incredible and cool it was much more I
think finicky and bespoke at the early
stages when the discovery was first made
but at this point it's become almost
industrialized there are what's called
contract research organizations vendors
that will take a sample from a human and
reverted back to stem cell status and it
works a very good fraction of the time
now there are people who will ask I
think good questions is this really
truly a stem cell er doesn't remember
certain aspects of what of changes that
were made in the human beyond the
genetics it's fast as a skin cell yeah
it's fast as a skin cell or its past in
terms of exposures to different
environmental factors and so on so I
think the consensus right now is that
these are not always perfect and there
is a little bits and pieces of memory
sometimes but by and large these are
actually pretty good so one of the key
things well maybe maybe you can correct
me but one of the useful things for
machine learning is size scale of data
how easy it is to do these kinds of
reversals to stem cells and then disease
in a dish models at scale is this that a
huge challenge or or not so the reverse
the reversal is not as of this point
something that can be done at the scale
of tens of thousands or hundreds of
thousands I think total number of stem
cells or iPS cells that are what's
called induced pluripotent stem cells in
the world I think is somewhere between
five and ten thousand last I looked now
again that might not count things that
exist in this or that academic center
and they may add up to a bit more but
that's about the range so it's not
something that you could this point
generate IPS cells from a million people
but maybe you don't need to because
maybe that background is enough because
it can also be now perturbed in
different ways and some people have done
really interesting experiments in for
instance taking cells from a healthy
human and then introducing a mutation
into it using some of the using one of
the other miracle technologies that's
emerged last decade which is CRISPR gene
editing and introduced mutation that is
known to be pathogenic and so you can
now look at the healthy cells and
unhealthy cells the one with the
mutation and do a one-on-one comparison
where everything else is held constant
and so you could really start to
understand specifically what the
mutation does at the cellular level so
the IPS cells are a great starting point
and obviously more diversity is better
because you also want to capture ethnic
background and how that affects things
but maybe you don't need one from every
single patient with every single type of
disease because we have other tools at
our disposal well how much difference is
there between people I mentioned ethnic
background in terms of IPS cells so
we're all like it seems like these
magical cells that can do it to create
anything between different populations
different people is there a lot of
variability between stem cells well
first of all there's the variability
that's driven simply by the fact that
genetically we're different so a stem
cell let's drive for my genotype is
gonna be different from itself stem
cells derive from your genotype there's
also some differences that I have more
to do with for whatever reason
some people stem cells differentiate
better than other people stem cells we
don't entirely understand why so there's
certainly some differences there as well
but the fundamental difference and the
one that we really care about and is a
positive is that the is the fact that
the genetics are different and therefore
we capitulate my disease burden versus
your disease burden what's the disease
burden well it disease burden is just if
you think I mean it's not a well-defined
mathematical term although there are
mathematical formulations of it
it if you think about the fact that some
of us are more likely to get a certain
disease than others because we have more
variations in our genome that are
causative of the disease maybe fewer
that are protective of the disease
people have quantified that using what
are called polygenic risk scores which
look at all of the variations in an
individual person's genome and add them
all up in terms of how much risk they
confer for a particular disease and then
they've put people on a spectrum of
their disease risk and for certain
diseases where we've been sufficiently
powered to really understand the
connection between the many many small
variations that give rise to an
increased disease risk there is some
pretty significant differences in terms
of the risk between the people say at
the highest decile of this polygenic
risk score and the people at the lowest
decile sometimes those other differences
are you know factor of 10 or 12 higher
so there's definitely a lot that our
genetics contributes to disease risk
even if it's not by any stretch the full
explanation and from the machine
learning perspective their signal there
there is definitely signal in the
genetics and there is even more signal
we believe in looking at the cells that
are derived from those different
genetics because in principle you could
say all the signal is there the at the
genetics level so we don't need to look
at the cells but our understanding of
the biology so limited at this point
then seeing what actually happens at the
cellular level is a heck of a lot
closer to the human clinical outcome
than looking at the genetics directly
and so we can learn a lot more from it
than we could by looking at genetics
alone so just to get a sense that enough
it's easy to do but what kind of data is
useful in this disease in a dish model
like what what are what's what's the
source of raw data information and also
for my outsider's perspective sort of
biology and cells are squishy things and
I think they are how do you connect
literally you connect the computer to to
that which sensory mechanisms I guess so
that's another one of those revolutions
that have happened the last ten years
and that our ability to measure cells
very quantitatively has also
dramatically increased so back when I
started doing biology and you know late
90s early 2000s that was the initial era
where we started to measure biology in
really quantitative ways using things
like microarrays where you would measure
in a single experiment the activity
level what's called expression level of
multiple of every gene in the genome in
that sample and that ability is what
actually allowed us to even understand
that there are molecular subtypes of
diseases like cancer where up until that
point is like oh you have breast cancer
but then we looked we looked at the
molecular data it was clear that there's
different subtypes of breast cancer that
at the level of gene activity look
completely different to each other so
that was the beginning of this process
now we have the ability to measure
individual cells in terms of their gene
activity using what's called single cell
RNA sequencing which basically sequences
the RNA which is that activity level of
different genes for every gene in the
genome and you could do that at single
cell level so that's an incredibly
powerful way of measuring cells I mean
you literally count the number of
transcripts oh really turns that squishy
thing in something that's digital
another tremendous
this data source that's emerged the last
few years is microscopy and and
specifically even super resolution
microscopy where you could use digital
reconstruction to look at sub cellular
structures sometimes even things that
are below the diffraction limit of light
by doing a sophisticated reconstruction
and again that gives you tremendous
amount of information at the sub
cellular level there's now more and more
ways that an amazing scientists out
there are developing for getting new
types of information from even single
cells and so that is a way of turning
those squishy things into digital data
into beautiful datasets but so that data
said then with machine learning tools
allows you to maybe understand the
developmental like the mechanism of the
a particular disease and if it's
possible to sort of at a high level
describe how does how does that help
lead to drug discovery that can help
prevent reverse that mechanism so I
think there's different ways in which
this data could potentially be used some
people use it for scientific discovery
and say oh look we see this phenotype at
the cellular level so let's try and work
our way backwards and think which genes
might be involved in pathways that give
rise that so that's a very sort of
analytical method to sort of work our
way backwards using our understanding of
known biology some people use it in a
somewhat more you know sort of forward
that would if that was a backward this
would be forward which is to say okay if
I can perturb this gene doesn't show a
phenotype that is similar to what I see
in disease patients and so maybe that
gene is actually causal of the disease
so that's a different way and then
there's what we do which is basically to
take that very large collection of the
and use machine learning to uncover the
patterns that emerge from it so for
instance what are those subtypes that
might be similar at the human clinical
outcome but quite distinct when you look
at the molecular data and then if we can
identify such a subtype are there
interventions that if I apply it to
cells that come from this subtype of the
disease and you apply that intervention
it could be a drug or it could be a
CRISPR gene intervention it does it
revert the disease state to something
that looks more like normal happy
healthy cells and so hopefully if you
see that that gives you a certain hope
that that intervention will also have a
meaningful clinical benefit to people
and there's obviously a bunch of things
that you would want to do after that to
validate that but it's a very different
and much less hypothesis-driven way of
uncovering new potential interventions
and might give rise to things that are
not the same things that everyone else
is already looking at that's uh I don't
know I'm just like to psychoanalyze my
own feeling about our discussion
currently it's so exciting to talk about
so if I'm Ashiya fundamentally well
something that's been turned into a
machine learning problem and that says
can have so much real-world impact
that's kind of exciting because I'm so
most of my days spent with datasets that
I guess closer to the news groups okay
so this is a kind of it just feels good
to talk about in fact I don't almost
don't want to talk about machine
learning I want to talk about the
fundamentals of the data set which is
which is an exciting place to be I agree
with you it's what gets me up in the
morning it's also what attracts a lot of
the people who work at in seat row two
in seat row because I think all of the
certainly all of our machine learning
people are outstanding and could go get
a job you know selling ads online or
doing commerce or even self-driving cars
yes but but I think they would want they
they come to us because what because
they want to work on something that
more of an aspirational nature and can
really benefit humanity what with these
with these approaches what do you hope
what kind of diseases can be helped we
mentioned Alzheimer said schizophrenia
type 2 diabetes can you just describe
the various kinds of diseases that this
approach can it can help well we don't
know and I try and be very cautious
about making promises about some things
that o we will cure X that people make
that promise and I think it's I tried to
first deliver and then promise as
opposed to the other way around there
are characteristics of a disease that
make it more likely that this type of
approach can potentially be helpful so
for instance diseases have a very strong
genetic basis are ones that are more
likely to manifest and a stem cell
derived model we would want the cellular
models to be relatively reproducible and
robust so that you could actually get a
enough of those cells and in a way that
isn't very highly variable and noisy you
would want the disease to be relatively
contained in one or a small number of
cell types that you could actually
create in an in vitro in a dish setting
whereas if it's something that's really
broad and systemic and involves multiple
cells that are in very distal parts of
your body putting that all in the dish
is really challenging so we want to
focus on the ones that are most likely
to be successful today with the hope I
think that it's really smart
bioengineers out there are developing
better and better systems all the time
so the diseases that might not be
tractable today might be tractable in
three years so for instance five years
ago these stem cell drive models didn't
really exist people were doing most of
the work in cancer cells and the cancer
cells are very very poor models of most
human biology because they're a they
were cancer to begin with and B as you
passage them and they proliferate in a
dish they become because of the genomic
instability even less
similar to human biology now we have
these stem cell derived models we have
the capability to reasonably robustly
not quite at the right scale yet but
close to derive what's called organoids
which are these teeny little sort of
multicellular organ of an organ system
so there's cerebral organoids and liver
organoids and kidney organoids and yeah
brain organize organize possibly the
coolest thing I've ever seen and then I
think we're starting to see things like
connecting these organize to each other
so that you could actually start and
there's some really cool papers that
start to do that where you can actually
start to say okay can we do multi organ
system stuff there's many challenges
that it's not easy by any stretch but it
might I'm sure people will figure it out
and in three years or five years there
will be disease moles that we could make
for things that we can't make today yeah
and this conversation would seem almost
outdated with a kind of scale that could
be achieved in like three years
that would be so cool the you've
co-founded Coursera with injurying and
were part of the whole MOOC revolution
so to jump topics a little bit can you
maybe tell the origin story of the
history the origin story of MOOCs of
Coursera and in general the your
teaching to huge audiences on a very
sort of impactful topic of AI general so
I think the origin story of MOOCs
emanates from a number of efforts that
occurred at Stanford University around
you know the late 2000s where different
individuals within Stanford myself
included were getting really excited
about the opportunities of using online
technologies as a way of achieving both
improved quality of teaching and also
improved scale and so Andrew for
instance led the the
for engineering everywhere which was
sort of an attempt to take ten Stanford
courses and put them online just as you
know video lectures I led an effort
within Stanford to take some of the
courses and really create a very
different teaching model that broke
those up into smaller units and had some
of those embedded interactions and and
so on which got a lot of support from
University leaders because they felt
like it was potentially a way of
improving the quality of instruction in
Stanford by moving to what's now called
the flipped classroom model and so those
efforts eventually sort of started to
interplay with each other and created a
tremendous sense of excitement and
energy within the Stanford community
about the potential of online teaching
and led in the fall of 2011 to the
launch of the first inferred MOOCs by
the way MOOCs it's probably impossible
that people don't know but I guess
massive open online courses but online
courses so they're not come up with the
acronym I'm not particularly fond of the
acronym but it is what it is where this
Big Bang is not a great term for the
start of the universe but it is what it
is probably so anyway we so those
courses launched in in the fall of 2011
and there were within a matter of weeks
with no real publicity campaign just a
New York Times article that went viral
about a hundred thousand students or
more in each of those courses and I
remember this conversation that Andrew
and I had was like wow just there's this
real need here and I think we both felt
like sure we were accomplished academics
and we could go back and you know go
back to our lives write more papers but
if we did that then this wouldn't happen
and it seemed too important not to
happen and so we spent a fair bit of
time debating do we want to do this as a
Stanford efforts kind of building on
what we'd started do we want to do this
as a for-profit company doing this is a
non-profit and we decided
ultimately to do it as we did with
Coursera and so you know we started
really operating as a company at the
beginning of 2012 but how did you was
that really surprising to you how how do
you at that how did you at that time and
at this time make sense of this need for
sort of global education you mentioned
that you felt that while the the
popularity indicates that there's a
hunger for sort of globalization of
learning I think there is a hunger for
learning that you know globalization is
part of it but I think it's just a
hunger for learning the world has
changed in the last 50 years it used to
be that you finished college you got a
job by and large the skills that you
learned in college were pretty much what
got you through the rest of your job
history and and yeah you learned some
stuff but it wasn't a dramatic change
today we're in a world where the skills
that you need for a lot of jobs they
didn't even exist when you went to
college and the jobs and many of the
jobs that exist when you went the
college don't even exist today or dying
so part of that is due to AI but not
only and we need to find a way of
keeping people giving people access to
the skills that they need today and I
think that's really what's driving a lot
of this hunger so I think if we even
take a step back all for you all the
start in trying to think of new ways to
teach or to you know new ways to sort of
organize the material and present the
material in a way that would help the
education process the better gotcha yeah
so what have you learned about effective
education from this process of playing
of experimenting with different ideas so
we learned a number of things some of
which I think could translate back and
have translated back effectively to how
people teach on campus and some of which
I think are more specific to people who
learn online
and more sort of people who learn as
part of their daily life so we learned
for instance very quickly that short is
better so people who are especially in
the workforce can't do a 15-week
semester long course they just can't fit
that into their lives
shortly can you uh can you describe the
shortness of what the the the entirety
so every aspects of the little lecture
short this the less your short
the course is short both we started out
you know the first online education
efforts were actually mi t--'s
OpenCourseWare initiatives and that was
you know recording of classroom lectures
and you know hour and a half or
something like that yeah that didn't
really work very well I mean some people
benefit I mean of course they did but
it's not really very palatable
experience for someone who has a job and
you know three kids and that they need
to run errands and such they can't fit
15 weeks into their life and and the
hour and a half is really hard so we
learned very quickly and we started out
with short video modules and over time
we made them shorter because we realized
that 15 minutes was still too long if
you want to fit in when you're waiting
in line for your kids doctor's
appointment it's better if it's 5 to 7
we learned that 15 week courses don't
work and you really want to break this
up into shorter units so that there is a
natural completion point gives people a
sense of they're really close to
finishing something meaningful they can
always come back and take part two and
part three we also learned that
compressing the content works really
well because if some people that pace
works well for others they can always
rewind and watch again and so people
have the ability to then learn at their
own pace and so that flexibility the the
brevity and the flexibility are both
things that we found to be very
important we learned that engagement
during the content is important and the
quicker you give people feedback the
more likely they are to be engaged
hence the introduction of these which we
actually was an intuition that I had
going in and and
was then validated using data that
introducing some of these sort of little
quick micro quizzes into the lectures
really helps self graded as
automatically graded assessments really
help too because it gives people
feedback see there you are so all these
are valuable and then we learn about two
other things - oh we did some really
interesting experiments for instance on
though gender bias and how having a
female role model as an instructor can
change the balance of men to women in
terms of especially in stem courses and
you could do that online by doing a/b
testing in ways that would be really
difficult to go on campus oh that's
exciting but so the shortness the
compression I mean that's actually so
that that probably is true for all you
know good editing is always just
compressing the content making it
shorter so that puts a lot of burden on
the creator of the the instructor and
the creator of the educational content
probably most lectures at MIT or
Stanford could be five times shorter if
the preparation was put was put enough
so maybe people might disagree with that
but like the Christmas the clarity that
a lot of them like Coursera delivers is
how much effort does that take so first
of all let me say that it's not clear
that that crispness would work as
effectively and a face-to-face setting
because people need time to absorb the
material and so you need to at least
pause and give people a chance to
reflect that maybe practice and that's
what MOOCs do is that they give you
these chunks of content and then ask you
to practice with it and that's where I
think some of the newer pedagogy that
people are adopting and face-to-face
teaching they have to do with
interactive learning and such it can be
really helpful but both those approaches
whether you're doing that type of
methodology and online teaching or in
that flipped classroom interactive
teaching what site applause what's
flipped classroom flipped classroom is a
way in which online content is
used a supplement face-to-face teaching
where people watch the videos perhaps
and do some of the exercises before
coming to class and then when they come
to classes actually to do much deeper
problem solving oftentimes in a group
but any one of those different
pedagogy's that are beyond just standing
there and droning on in front of the
classroom for an hour and 15 minutes
require a heck of a lot more preparation
and so it's one of the challenges I
think that people have that we had when
trying to convince instructors to teach
on Coursera and it's part of the
challenges that pedagogy experts on
campus have in trying to get faculty to
teach differently is that it's actually
harder to teach that way than it is to
stand there drone do you think MOOCs
will replace in-person education or
become the majority of in-person of
Education of the way people learn in the
future again the future could be very
far away but where's the trend going do
you think so I think it's a nuanced and
complicated answer I don't think MOOCs
will replace face-to-face teaching I
think learning is in many cases a social
experience and even at Coursera we had
people who naturally formed study groups
even when they didn't have to just come
and talk to each other and we found that
that actually benefited their learning
in very important ways so there was more
success in among learners who had those
study groups than among ones who didn't
so I don't think it's just gonna oh
we're all gonna just suddenly learn
online with a computer and no one else
in the same way that you know recorded
music has not replaced live concerts but
I do think that especially when you are
thinking about continuing education the
stuff that people get when they're
traditional whatever high school college
education is done and they yet have to
maintain their level of expertise and
skills in a rapidly changing world I
think people will
sooo more and more educational content
in this online format because going back
to school for formal education is not an
option for most people
briefly I know it might be a difficult
question to ask but there's a lot of
people fascinated by artificial
intelligence by machine learning but
deep learning is there a recommendation
for the next year or for a lifelong
journey as somebody interested in this
how do they how do they begin how do
they enter that learning journey I think
the important thing is first to just get
started and there's plenty of online
content that one can get for both the
core foundations of mathematics and
statistics and programming and then from
there to machine learning I would
encourage people not to skip too quickly
past the foundations because I find that
there is a lot of people who learn
machine learning whether it's online or
on campus without getting those
foundations and they basically just turn
the crank on existing models in ways
that they don't allow for a lot of
innovation and an adjustment to the
problem at hand but also be or sometimes
just wrong and they don't even realize
that their application is wrong because
there's artifacts that they haven't
fully understood so I think the
foundations machine learning is an
important step and then and then
actually start solving problems try and
find someone to solve them with because
especially at the beginning is useful to
have someone to bounce ideas off and fix
mistakes that you make and and you can
fix mistakes that they make but but then
just find practical problems whether
it's in your workplace or if you don't
have that catechol competitions or such
are a really great place to find
interesting problems and just practice
practice perhaps a bit of a romanticized
question but what idea in deep learning
do you find have you found in your
journey the most beautiful or surprising
or interesting
perhaps not just deep learning but AI in
general statistics good answer with two
things one would be the foundational
concept of end to end training which is
that you start from the raw data and you
train something that is not like a
single piece but rather the towards the
actual goal that you're looking to from
the raw data to the outcome like and
nothing no no details in between well
not no details but the fact that you I
mean you could certainly introduce
building blocks that were trained
towards other tasks and actually coming
to that in my second half of the answer
but it doesn't have to be like a single
monolithic blob in the middle actually I
think that's not ideal but rather the
fact that at the end of the day you can
actually train something and goes all
the way from the beginning to the end
and the other one that I find really
compelling is the notion of learning a
representation that in its turn even if
it was trained to another task can
potentially be used as a much more rapid
starting point to solving a different
task and that's I think reminiscent of
what makes people successful learners
it's something that is relatively new in
the machine learning space I think it's
underutilized even relative to today's
capabilities but more and more of how do
we learn sort of reusable representation
so end to end and transfer learning yeah
is it surprising to you that neural
networks are able to in many cases do
these things it says it may be taking
back to when you when you first would
dive deep into neural networks or in
general even today is it surprising that
neural networks work at all and work
wonderfully to do this kind of raw and
then learning and even transfer learning
I think I was surprised by how
well when you have large enough amounts
of data it's possible to find a
meaningful representation in what is an
exceedingly high dimensional space and
so I find that to be really exciting and
people are still working out the math
for that there's more papers on that
every year and I think it's would be
really cool if we figured that out but
that to me was a surprise because in the
early days when I was starting my weigh
in machine learning and the data sets
were rather small I think we we believed
I believe that you needed to have a much
more constrained and knowledge rich
search space to really make to really
get to a meaningful answer and I think
it was true at the time what I think is
is still a question is will a completely
knowledge free approach where there's no
prior knowledge going into the
construction of the model is that going
to be the solution or not it's not
actually the solution today in the sense
that the architecture of a you know
convolutional neural network that's used
for images is actually quite different
to the type of networks it's used for
language and yet different from the one
that's used for speech or biology or any
other application there's still some
insight that goes into the structure of
the network to get the the right
performance will you be able to come up
with the universal learning machine I
don't know I wonder if there's always
has to be some insight injected
somewhere or whether it can converge so
you've done a lot of interesting work
with probabilistic graphical models in
general Bayesian deep learning and and
so on so can you maybe speak high level
how can learning systems deal with
uncertainty one of the limitations I
think of a lot of machine learning
models is that
they come up with an answer and you
don't know how much you can believe that
answer and oftentimes the the the answer
is actually quite poorly calibrated
relative to its uncertainties even if
you look at where the um you know the
the the confidence that comes out of the
say the neural network at the end and
you ask how much more likely is an
answer of zero point eight versus zero
point nine it's not really in any way
calibrated to the to the actual
reliability of that network and how true
it is and the further away you move from
the training data the more not only the
more wrong then that workers often is
more wrong and more confident in a
strong answer and that is a serious
issue in a lot of application areas so
when you think for instance about
medical diagnosis as being maybe an
epitome of how problematic this can be
if you were training your network on a
certain set of patients on a certain
patient population and I have a patient
that is an outlier and there's no human
that looks at this and that patient is
put into a neural network in your
network not only gives a completely
incorrect diagnosis but it's supremely
confident and it's wrong answer you
could kill people so I think creating
more of an understanding of how do you
do snut works that are calibrated in our
uncertainty and can also say you know I
give up I don't know what to say about
this particular data instance because
I've never seen something that
sufficiently liked it before I think
it's going to be really important in
mission-critical applications especially
ones where human life is at stake and
that includes the you know medical
applications but it also includes you
know automated driving because you'd
want the network to be able to you know
what I have no idea what this blob is
that I'm seeing in the middle of the
rest I'm just gonna stop because I don't
want to potentially run over a
pedestrian that I don't recognize is
there good mechanisms ideas of how to
allow
learning systems to provide that
uncertainty whatever along with their
predictions certainly people have come
up with mechanisms that involve Bayesian
deep learning deep learning that
involves Gaussian processes I mean there
is a slew of different approaches that
people have come up with
there's methods that use ensembles of
networks with trained with different
subsets of theta or different random
starting points those are actually
sometimes surprisingly good at creating