Kind: captions
Language: en
the following is a conversation with
Christos Kudrow vice president of
engineering at Google and head of search
and discovery at YouTube also known as
the YouTube algorithm YouTube has
approximately 1.9 billion users and
every day people watch over 1 billion
hours of YouTube video it is the second
most popular search engine behind Google
itself for many people it is not only a
source of entertainment but also how we
learn new ideas from math and physics
videos to podcasts to debates opinions
ideas from out-of-the-box thinkers and
activists some of the most tense
challenging and impactful topics in the
world today YouTube and other content
platforms receive criticism from both
viewers and creators as they should
because the engineering task before them
is hard and they don't always succeed
and the impact of their work is truly
world-changing to me YouTube has been an
incredible wellspring of knowledge I've
watched hundreds if not thousands of
lectures that changed the way I see many
fun about those ideas in math science
engineering and philosophy but it does
put a mirror to ourselves and keeps the
responsibility of the steps we take in
each of our online educational journeys
into the hands of each of us the YouTube
algorithm has an important role in that
journey of helping us find new exciting
ideas to learn about that's a difficult
and an exciting problem for an
artificial intelligence system as I've
said in lectures and other forums
recommendation systems will be one of
the most impactful areas of AI in the
21st century
and YouTube is one of the biggest
recommendation systems in the world this
is the artificial intelligence podcast
if you enjoy it subscribe on YouTube
give it five stars an Apple podcast
follow on Spotify supported on patreon
or simply connect with me on Twitter
Alex Friedman spelled Fri D ma a.m. I
recently started doing ads at the end of
the introduction I'll do one or two
minutes after introducing the episode
and never any ads in the middle that can
break the flow of the conversation
I hope that works for you and doesn't
hurt the listening experience this show
is presented by cash app the number one
finance app in the App Store I
personally use cash app to send money to
friends but you can also use it to buy
sell and deposit Bitcoin in just seconds
cash app also has a new investing
feature you can buy fractions of a stock
say $1 worth no matter what the stock
price is brokerage services are provided
by cash app investing a subsidiary of
square and member si PC I'm excited to
be working with cash app to support one
of my favorite organizations called
first best known for their first
robotics and Lego competitions they
educate and inspire hundreds of
thousands of students in over 110
countries and have a perfect rating and
Charity Navigator
which means that donated money is used
to maximum effectiveness when you get
cash app from the App Store Google Play
and use code Lex podcast you'll get ten
dollars in cash app will also donate ten
dollars to the first which again is an
organization that I've personally seen
inspire girls and boys the dream of
engineering a better world and now
here's my conversation with Christos
Gaudreau YouTube is the world's second
most popular search engine behind Google
of course we watch more than 1 billion
hours of YouTube videos a day more than
Netflix and facebook video combined
YouTube creators upload over 500
thousand hours of video every day
average lifespan of a human being just
for comparison is about 700,000 hours so
what's uploaded every single day is just
enough for a human to watch in a
lifetime so let me ask an absurd
philosophical question if from birth
when I was born and there's many people
born today with the internet I watched
YouTube videos non-stop do you think
there are trajectories through YouTube
video space that can maximize my average
happiness or maybe education or my
growth as a human being I think there
are some great trajectories through
YouTube videos but I wouldn't recommend
that anyone
and all of their waking hours or all of
their hours watching YouTube I mean I
think about the fact that YouTube has
been really great for my kids for
instance my oldest daughter you know
she's been watching YouTube for several
years she watches Tyler Oakley and the
vlogbrothers and I know that it's had a
very profound and positive impact on her
character and my younger daughter she's
a ballerina and her teachers tell her
that YouTube is a huge advantage for her
because she can practice a routine and
watch like professional dancers do that
same routine and stop it and back it up
and rewind and all that stuff right so
it's been really good for them and then
even my son is a sophomore in college he
he got through his linear algebra class
because of a channel called three blue
one brown which you know helps you
understand linear algebra but in a way
that would be very hard for anyone to do
on a whiteboard or a chalkboard and so I
think that those experiences from my
point of view were very good and and so
I can imagine really good trajectories
through YouTube yes have you looked at
do you think of broadly about that
trajectory over a period cuz YouTube was
growing up now so over a period of years
you just kind of gave a few anecdotal
examples but you know I used to watch
certain shows on YouTube I don't anymore
I've moved on to other shows and
ultimately you want people to from
YouTube's perspective to stay on YouTube
to grow as human beings on YouTube so
you have to think not just what makes
them engage today or this month but also
over a period of years absolutely that's
right I mean if YouTube is going to
continue to enrich people's lives then
you know then it has to grow with them
and and people's interests change over
time and so I think we've we've been
working on this problem and I'll just
say it broadly is like how to introduce
diversity and introduce people who are
watching one thing to something else
they might like
we've been working on that problem all
the eight years I've been at YouTube
it's a hard problem because I mean of
course it's trivial to introduce
diversity that doesn't help a random
video I could just randomly select a
video from the billions that we have
it's likely not to even be in your
language so haha the likelihood that you
would watch it and develop a new
interest is very very low and so what
you want to do when you're trying to
increase diversity is find something
that is not too similar to the things
that you've watched but also something
that you might be likely to watch and
that balance finding that spot between
those two things is quite challenging so
the diversity of content diversity of
ideas it's uh it's a really difficult
it's the thing like that's almost
impossible to define alright like what's
different so how do you think about that
so two examples is a I'm a huge fan of
three blue one Brown say and then one
diversity I wasn't even aware of a
channel called veritasium width which is
a great science physics whatever channel
so one version of diversity is showing
me Derek's veritasium channel which I
was really excited to discover actually
now watch a lot of his videos okay so
you're a person who's watching some math
channels and you might be interested in
some other science or math channels so
like you mentioned the first kind of
diversity is just show you some some
things from other channels that are
related but not just you know not all
the three blue one Brown Channel throw
in a couple others so so that's the
maybe the first kind of diversity that
we started with many many years ago
taking a bigger leap is is about I mean
the mechanisms we do we use for that is
is we basically cluster videos and
channels together mostly videos we do
every almost everything at the video
level and so we'll we'll make some kind
of a cluster
some embedding process and then and then
measure you know what is the likelihood
that a that users who watch one cluster
might also watch another cluster that's
very distinct so we may come to find
that that people who watch science
videos also like jazz this is possible
right and so and so because of that
relationship that we've identified
through the through the embeddings and
then the measurement of the people who
watch both we might recommend a jazz
video once in a while so there's this
clustering the embedding space of jazz
videos and science videos and so you
kind of try to look at aggregate
statistics where if a lot of people that
jump from science cluster to the jazz
cluster tend to remain as engaged or
become more engaged then that's that
means those two are they should hop back
and forth and they'll be happy right
there's a higher likelihood that a
person from who's watching science would
like jazz then the person watching
science would like I don't know backyard
railroads or something else right and so
we can try to measure these likelihoods
and use that to make the best
recommendation we can so okay so we'll
talk about the machine learning of that
but I have to linger on things that
neither you or anyone have an answer to
there's gray areas of truth which is for
example now I can't believe I'm going
there but politics it it happens so that
the certain people believe certain
things and they're very certain about
them let's move outside the red versus
blue
politics of today's world but there's
different ideologies for example in
college I read quite a lot of iron rand
i studied and that's a particular
philosophical ideologies I find I found
it interesting to explore okay so that
was that kind of space I've kind of
moved on from that cluster
intellectually but it nevertheless is an
interesting cluster there's I was born
in Soviet Union socialism communism is a
certain kind of political idea
that's really interesting to explore
again objectively just there's a set of
beliefs about how the economy should
work and so on and so it's hard to know
what's true or not in terms of people
within those communities they're often
advocating that this is how we achieve
utopia in this world and they're pretty
certain about it so how do you try to
manage politics in this chaotic divisive
world not positively any kind of ideas
in terms of filtering what people should
watch next and in terms of also not
letting certain things be on YouTube
this is exceptionally difficult
responsibility right well the
responsibility to get this right is our
top priority and and the first comes
down to making sure that we have good
clear rules of the road right like just
because we have freedom of speech
doesn't mean that you can literally say
anything right like we as a society have
accepted certain restrictions on our
freedom of speech there are things like
libel laws and things like that and so
where we can draw a clear line we do and
we continue to evolve that line over
time however as you point it out
wherever you draw the line there's going
to be a border line and in that border
line area we are going to maybe not
remove videos but we will try to reduce
the recommendations of them or the
proliferation of them by demoting them
and then alternatively in those
situations try to raise what we would
call Thorat ativ or credible sources of
information so we're not trying to I
mean you mentioned Iran and communism
you know those are those are two like
valid points of view that people are
going to debate and discuss and and of
course people who believe and one of the
other of those things are going to try
to persuade other people to their point
of view and so we're not trying to
settle that
choose a side or anything like that what
we're trying to do is make sure that the
the people who are expressing those
point of view and and offering those
positions are authoritative and credible
so let me ask a question
about people I don't like personally you
heard me I don't care if you leave
comments on this is uh and but sometimes
there's brilliantly funny which is
trolls so it's people who kind of mock I
mean the Internet is full the reddit of
mock style comedy where people just kind
of make fun of point out that the
emperor has no clothes and there's
brilliant comedy and that but sometimes
it can get cruel and mean so on that on
the mean point and sorry to linger on
these things that have no good answers
but it actually is I totally hear you
that this is really important you're
trying to solve it but how do you reduce
the meanness of people on YouTube I
understand that anyone who uploads
YouTube videos has to become resilient
to a certain amount of meanness like
I've heard that from many creators and
we would we are trying in various ways
comment ranking allowing certain
features to block people to reduce or or
make that that meanness or that trolling
behavior less effective on YouTube yeah
and so I mean it's it's very important
but it's something that we're we're
gonna keep having to work on and and you
know as we improve it maybe we'll get to
a point where where people don't have to
suffer this sort of meanness when they
upload YouTube videos I hope we do but
you know but it just does seem to be
something that you have to be able to
deal with as a YouTube creator now it is
do you have a hope that he mentioned two
things that can I agree was so there's a
machine-learning approach of ranking
comments based on whatever based on how
much they contribute to the healthy
conversation let's put it that way then
the other is almost an interface
question of how do you how does the
Creator filter so block or how does how
do humans themselves the users of
YouTube manage their own conversation do
you have hope that these two tools will
create a better society without limiting
freedom of speech too much without sort
of Antonin even like saying that people
what do you mean limiting sort of
curating speech I mean I think that that
overall is our whole project here at
YouTube right like yeah we fundamentally
believe and I personally believe very
much that YouTube can be great it's been
great for my kids I think it can be
great for society but it's absolutely
critical that we get this responsibility
part right and that's why it's our top
priority Susan Wojcicki who's the CEO of
YouTube she says something that I
personally find very inspiring which is
that we want to do our jobs today in a
manner so that people 20 and 30 years
from now will look back and say you know
YouTube they they really figured this
out they really found a way to strike
the right balance between the openness
and the value that the openness has and
also making sure that we are meeting our
responsibilities to users in society so
the burden on YouTube actually is quite
incredible and the one thing that people
don't I don't give enough credit to the
seriousness and the magnitude of the
problem I think so I I personally hope
that you do solve it because a lot is in
your hand a lot is riding on your
success or failure so it's besides of
course running a successful company
you're also curating the content of the
internet and the conversation the
internet that's a powerful thing so one
thing that people wander about is how
much of it
can be solved with pure machine learning
so looking at the data studying the data
and creating algorithms that curate the
comments curate the content and how much
of it needs human intervention meaning
people here YouTube in a room sitting
and thinking about what is the nature of
truth what is what are the ideals that
we should be promoting that kind of
thing so algorithm versus human input
what's your sense I mean my own
experience has demonstrated that you
need both of those things algorithms I
mean you're familiar with machine
learning algorithm and the thing they
need most is data and the data is
generated by humans and so for instance
when we're building a system to try to
figure out which are the videos that are
misinformation or borderline policy
violations well the first thing we need
to do is get human beings to make
decisions about which which of those
videos are in which category and then we
use that data and and basically you know
take that information that's that's
determined and governed by humans and
and extrapolated or apply it to the
entire set of billions of YouTube videos
and we couldn't we we couldn't get to
all the videos on YouTube well without
the humans and we we couldn't use the
humans to get to all the videos of
YouTube so there's no world in which you
have only one or the other of these
things and just as you said a lot of it
comes down to people at YouTube spending
a lot of time trying to figure out what
are the right policies you know what are
the outcomes based on those policies are
they the kinds of things we want to see
and then once we kind of get a get an
agreement or or build some consensus
around around what the policies are well
then we've got to find a way to
implement those policies across all of
YouTube and that's where both the human
beings we call them evaluators or
reviewers come into play to help us with
that and then and then once we get a lot
of training data from them
then we apply the machine learning
techniques to take it even further do
you have a sense that these human beings
have a bias in some kind of direction
sort of I mean that's the interesting
question we do sort of in autonomous
vehicles and computer vision in general
a lot of annotation and we rarely ask
what bias do the annotators have you
know the it even in the sense that
they're better than they're better
anting certain things and others for
example people are much better at
annotating segmentation at segmenting
cars in a scene versus segmenting bushes
or trees you know there's specific
mechanical reasons for that but also
because the cement its semantics gray
area and and just for a lot of reasons
people are just terrible at annotating
trees okay so in the same kind of sense
do you think of in terms of people
reviewing videos or annotating the
content of videos is there some kind of
bias that you're aware of or seek out in
that human input well we take steps to
try to overcome these kinds of biases or
biases that we think would be
problematic so for instance like we
asked people to have a bias towards
scientific consensus that's something
that we we instruct them to do we ask
them to have a bias towards
demonstration of expertise or
credibility or authoritative nests but
there are other biases that we that we
want to make sure to try to remove and
there's many techniques for doing this
one of them is you send the same thing
to be reviewed to many people and so you
know that's one technique another is
that you make sure that the people that
are doing these sorts of tasks are from
different backgrounds and different
areas of the United States
of the world but then even with all of
that it's possible for certain kinds of
what we would call unfair biases to
creep into machine learning systems
primarily as you said because maybe the
training data itself comes in in in a
biased way and so we also have worked
very hard on the improving the machine
learning systems to remove and reduce
unfair biases when it's when it goes
against or or has involved some
protected class for instance thank you
for exploring with me some of the more
challenging things I'm sure there's a
few more that we'll jump back to but let
me jump into the fun part which is maybe
the basics of the quote-unquote YouTube
algorithm what is the YouTube algorithm
look at to make recommendation for what
to watch next was from a machine
learning perspective or when you search
for a particular term how does it know
what to show you next because it seems
to at least for me do an incredible job
both well that's kind of you to say it
didn't used to do a very good job but
it's gotten better over the years even
even I observed that it's improved quite
a bit
those are two different situations like
when you search for something YouTube
uses the best technology we can get from
Google to make sure that that the
YouTube search system finds what
someone's looking for and of course the
very first things that one thinks about
is okay well does the word occur in the
title for instance you know but there
but there are much more sophisticated
things where we're mostly trying to do
some syntactic match or or maybe a
semantic match based on words that we
can add to the document itself for
instance you know maybe is is this video
watched a lot after this query right
that's something that
we can observe and then as a result make
sure that that that document would be
retrieved for that query now when you
talk about what kind of videos would be
recommended to watch next that's
something again we've been working on
for many years and probably the first
the first real attempt to do that
well was to use collaborative filtering
so you can't describe what collaborative
filtering is sure it's just basically
what we do is we observe which videos
get watched close together by the same
person and if you observe that and if
you can imagine creating a graph where
the videos that get watched close
together by the most people are sort of
very close to one another in this graph
and videos that don't frequently get
watch close too close together by the
same person or the same people are far
apart then you end up with this graph
that we call the related graph that
basically represents videos that are
very similar or related in some way and
what's amazing about that is that it
puts all the videos that are in the same
language together for instance and we
didn't even have to think about language
just does it yeah I didn't it puts all
the videos that are about sports
together and it puts most of the music
videos together and it puts all of these
sorts of videos together just because
that's sort of the way the people using
YouTube behave so that already cleans up
a lot of the problem it takes care of
the lowest hanging fruit which happens
to be a huge one of just managing these
millions of videos that's right I
remember a few years ago I was talking
to someone who was trying to propose
that we do a research project concerning
people who who are bilingual and this
person
was making this proposal based on the
idea that YouTube could not possibly be
good at recommending videos well to
people who are bilingual and so she was
telling me about this and I said well
can you give me an example of what
problem do you think we have on YouTube
with the recommendations and so she said
well I'm a researcher in in the US and
and when I'm looking for academic topics
I want to look I want to see them in
English and so she searched for one
found a video and then looked at the
watch next suggestions and they were all
in in English and so she said oh I see
YouTube must think that I speak only
English and so she said now I'm actually
originally from Turkey and sometimes
when I'm cooking let's say I want to
make some baklava I really like to watch
videos that are in Turkish and so she
searched for a video about making the
baklava and then and then selected it it
was in Turkish and the watch next
recommendations were in Turkish and she
just couldn't believe how this was
possible and how is it that you know
that I speak both these two languages
and put all the videos together and it's
just as a route come of this related
graph that's created through
collaborative filtering so for me one of
my huge interest is just human
psychology right and and that's such a
powerful platform on which to utilize
human psychology to discover what people
individual people want to watch next but
it's also be just fascinating to me
you know I've Google search has ability
to look at your own history and I've
done that before
just just what I've searched three years
for many many years and it's fascinating
picture of Who I am actually and I don't
think anyone's ever summarized that I
personally would love that a summary of
who I am as a person on the Internet to
me because I think it reveals I I think
it puts a mirror to me or to others you
know that's actually quite revealing and
interesting you know just maybe in the
number of it's a joke but not really is
the
of cat videos I've watched videos of
people falling you know stuff that's
absurd that kind of stuff it's really
interesting and of course it's really
good for the machine learning aspect to
to show to figure out what to show next
but it's interesting hey have you just
as a tangent played around with the idea
of giving a map to people sort of as
opposed to just using this information
to show us next showing them here are
the clusters you've loved over the years
kind of thing well we do provide the
history of all the videos that you've
watched yes so you can definitely search
through that and look through it and
search through it to see what it is that
you've been watching on YouTube we have
actually in various times experimented
with this sort of cluster idea finding
ways to demonstrate or show people what
topics they've been interested in or
what what clusters they've watched from
it's interesting that you bring this up
because in some sense the way the
recommendation system of YouTube sees a
user is exactly as the history of all
the videos they've watched on YouTube
and so you can think of yourself or any
user on YouTube as kind of like a DNA
strand of all your videos right that
sort of represents you you can also
think of it as maybe a vector in the
space of all the videos on YouTube and
so you know now once you think of it as
a vector in the space of all the videos
on YouTube then you can start to say
okay well you know which videos which
which other vectors are close to me and
to my vector and and that's one of the
ways that we generate some diverse
recommendations is because you're like
okay well you know these these people
seem to be closed with respect to the
videos they've watched on YouTube but
you know here's a topic or a video that
one of them has watched and enjoyed but
the other one hasn't that could be an
opportunity to make a good
recommendation I gotta tell you I mean I
know
for things that are impossible but I
would love to cluster than human beings
like I would love to know who has
similar trajectories as me you probably
would want to hang out alright there's a
social aspect there like actually
finding some of the most fascinating
people I find out in YouTube but have
like no followers and I start following
them and they create incredible content
and you know and on that topic I just
love to ask there's some videos just
blow my mind in terms of quality and
depth and just in every regard are
amazing videos and they have like 57
views okay how do you get videos of
quality to be seen by many eyes so the
measure of quality is it just something
yeah how do you know that something is
good well I mean I think it depends
initially on what sort of video we're
talking about so in the realm of let's
say you mentioned politics and news in
that realm you know quality news or
quality journalism relies on having a
journalism department right like you you
have to have actual journalists and
fact-checkers and people like that and
so in that situation and in others maybe
science or in medicine quality has a lot
to do with the authoritative nough sand
the credibility and the expertise of the
people who make the video now if you're
thinking about the other end of the
spectrum you know what is the highest
quality prank video for what is the
highest quality minecraft video yeah
right that might be the one that people
enjoy watching the most and watch to the
end or it might be the one that when we
ask people the next day after they
watched it were they satisfied with it
and so we in in especially in the realm
of entertainment have been trying to get
at better and better measures of quality
or satisfaction or enrichment
since I came to YouTube and we started
with well you know the first
approximation is the one that gets more
views but but you know we both know that
things can get a lot of views and not
really be that high quality especially
if people are clicking on something and
then immediately realizing that it's not
that great and abandoning it and that's
why we move from views to thinking about
the amount of time people spend watching
it what the premise that like you know
in some sense the time that someone
spends watching a video is related to
the value that they get from that video
it may not be perfectly related but it
has something to say about how much
value they get but even that's not good
enough
right because I myself have spent time
clicking through channels on television
late at night and ended up watching
under siege - for some reason I don't
know and if you were to ask me the next
day are you glad that you watched that
show on TV last night I'd say yeah I
wish I would have gone to bed or read a
book or almost anything else really and
so that's why some people got the idea a
few years ago to try to serve at users
afterwards and so so we get feedback
data from those surveys and then use
that in the machine learning system to
try to not just predict what you're
gonna click on right now what you might
watch for a while but what when we ask
you tomorrow you'll give four or five
stars - so just to summarize what are
the signals from a machine learning
perspective the user can provide he
mentions just clicking on the video
views the time watch maybe the relative
time watched the clicking like and
dislike on the video maybe commenting on
the video and those things all of those
things and then though the one I wasn't
actually quite aware of even though I
might have engaged in it is a survey
afterwards which is a brilliant idea is
there other signals all right I mean
that's already a really rich space of
signals to learn from is
something else well you mentioned
commenting also sharing the video if you
if you think it's worthy to be shared
with someone else you know within
YouTube or outside of YouTube as well
either
let's see you mentioned like dislike
yeah like and dislike how important is
that it's very important right we want
its predictive of satisfaction but it's
not it's not perfectly predictive
subscribe if you subscribe to the
channel of the person who made the video
then that also is a piece of information
and signals satisfaction although over
the years we've learned that people have
a wide range of attitudes about what it
means to subscribe we would ask some
users who didn't subscribe very much why
but they watched a lot from a few
channels we'd say well why didn't you
subscribe and they would say well I I
can't afford to pay for anything and you
know we tried to let them understand
like actually it doesn't cost anything
it's free it just helps us know that you
are very interested in this creator but
then we've asked other people who
subscribed to many things and and don't
really watch any of the videos from
those channels and we say well well why
did you subscribe to this if you weren't
really interested in any more videos
from that channel and they might tell us
why just you know I thought the person
did a great job and I just want to kind
of give him a high five yeah yeah and so
yeah that's where I I said I should
subscribe to channels where I just this
person is amazing I like this person but
then I like this person I really want to
support them that that's how I click
Subscribe right even though I may never
actually want to click on their videos
when they're releasing it I just love
what they're doing and it's maybe
outside of my interest area and so on
which is probably the wrong way to use
the subscribe button but I just want to
say congrats this is a great work well
so you have to deal with all the space
of people that see the subscribe button
it's totally different that's right and
so you know we we can't just close our
eyes and say
sorry you're using it wrong you know and
we're not gonna pay attention to what
you've done we need to embrace all the
ways in which all the different people
in the world use the subscribe button or
the like in the dislike button so in
terms of signals of machine learning
using for the search and for the
recommendation
you've mentioned title so like metadata
like text data that people provide
description and title and maybe keywords
so maybe you can speak to the value of
those things in search and also this
incredible fascinating area of the
content itself so the video content
itself trying to understand what's
happening in the video so YouTube would
release a dataset that you know the in
the machine learning and computer vision
world this is just an exciting space how
much is that currently how much he
playing with that currently how much is
your hope for the future of being able
to analyze the content of the video
itself well we have been working on that
also since I came to YouTube analyzing
the content analyzing the content while
video right
and what I can tell you is that our
ability to do it well is still somewhat
crude we can we can tell if it's a music
video
we can tell if it's a sports video we
can probably tell you that people are
playing soccer we probably can't tell
whether it's Manchester United or my
daughter's soccer team so these things
are kind of difficult and and using them
we can use them in some ways so for
instance we use that kind of information
to understand and inform these clusters
that I talked about and also maybe to
add some words like soccer for instance
to the video if if it doesn't occur in
the title or the description which is
remarkable that often it doesn't I one
of the things that I ask creators to do
is is please help us out with the title
in the description for instance we were
a a few years ago having a live stream
of some competition for World of
Warcraft
on YouTube
and it was a very important competition
but if you typed World of Warcraft in
search you wouldn't find it
well the Warcraft wasn't in the title
World of Warcraft wasn't in the title it
was match four seven eight you know a
team versus B team and World of Warcraft
wasn't the title yes like come on give
me being literal being literal on the
Internet is actually very uncool
which is the problem oh is that right
well I mean in some sense well some of
the greatest videos I mean there's a
humor to just being indirect being witty
and so on and actually being you know
machine learning algorithms want you to
be you know literal right usually want
to say what's in the thing be very very
simple and in in some sense that gets
away from wit and humor so you have to
play with both right so but you're
saying that for now sort of the content
of the title the content of the
description the actual text is is one of
the best ways to uh for the for the
algorithm to find your video and put
them in the right cluster that's right
and and I would go further and say that
if you want people human beings to
select your video in search then it
helps to have let's say World of
Warcraft in the title because why would
a person's you know if they're looking
at a bunch they type World of Warcraft
and they have a bunch of videos all of
whom say World of Warcraft except the
one that you uploaded well even the
person is gonna think well maybe this
isn't some house search made a mistake
this isn't really about World of
Warcraft so it's important not just for
the machine learning systems but also
for the people who might be looking for
this sort of thing they get a clue that
it's what they're looking for by seeing
that same thing prominently in the title
of the video okay let me push back on
that so I think from the algorithmic
perspective yes but if they typed in
World of Warcraft and saw a video that
with the title simply winning and and
and the thumbnail has like a sad orc or
something I don't know right like I
think that's much it's Iraq it gets your
curiosity up
and then if they could trust that the
algorithm was smart enough to figure out
somehow that this is indeed a World of
Warcraft video that would have created
the most beautiful experience I think in
terms of just the wit and the humor and
the curiosity that we human beings
actually have but you're saying I mean
realistically speaking is really hard
for the algorithm to figure out that the
content of that video will be a world of
warcraft and you have to accept that
some people are gonna skip it
yeah right I mean and so you're right
the people who don't skip it and select
it are gonna be delighted yeah but other
people say might say but yeah this is
not what I was looking for and making
stuff discoverable I think is what
you're really working on and hoping so
yeah so from your perspective to put
stuff in the description and remember
the collaborative filtering part of the
system it starts by the same user
watching videos together right so the
way that they're probably going to do
that is by searching for them that's a
fascinating aspect it's like ant
colonies that's how they find stuff is
so I mean you would agree for
collaborative filtering in general is
one curious ant
one curious user essential sort of just
a person who is more willing to click on
random videos and sort of explore these
cluster spaces in your sense how many
people are just like watching the same
thing over and over and over and over
and how many are just like the explorers
I just kind of like click on stuff and
then help help the other ant and the
ants colony discover the cool stuff do
you have a sense of that or no I really
don't think I have a sense me OK
relative sizes of those groups but I but
I would say that you know people come to
YouTube with some certain amount of
intent and as long as they to the extent
to which they they try to satisfy that
intent that certainly helps our systems
right because our systems rely on on
kind of a faithful amount of behavior
the right like and there are people who
try to trick us right there are people
and machines that try to associate
videos together that
really don't belong together but they're
trying to get that Association made
because it's profitable for them and so
we have to always be resilient to that
sort of attempt at gaming the system so
speaking to that there's a lot of people
that in a positive way perhaps I don't
know I I don't like it but I like to
gain want to try to gain the system to
get more attention and everybody
creators in a positive sense want to get
attention right so how do you how do you
work in this space when people create
more and more sort of click Beatty
titles and thumbnails sort of a very
tasking derek has made a video it
basically describes that it seems what
works is to create a high quality video
really good video what people would want
to watch and wants to click on it
but have clicked BT titles and
thumbnails to get him to click on it in
the first place and he's saying I'm
embracing this bactrim just gonna keep
doing it and I hope you forgive me for
doing it and you will enjoy my videos
once you click on them so in what sense
do you see this kind of clickbait style
attempt to manipulate to get people in
the door to manipulate the algorithm or
play with the algorithmic game the
algorithm I think that that you can look
at it as an attempt to game the
algorithm but even if you were to take
the algorithm out of it and just say ok
well all these videos happen to be lined
up which the algorithm didn't make any
decision about which one to put at the
top or the bottom but they're all lined
up there which one are the people going
to choose and and I'll tell you the same
thing that I told Derek is you know I
have a bookshelf and they have two kinds
of books on them science books I have my
math books from when I was a student and
they all look identical except for the
titles on the covers they're all yellow
they're all from Springer and they're
every single one of them the cover is
totally the same yes right yeah on the
other hand I have other more pop science
type books and they all have very
interesting covers right and they have
provocative
titles and things like that I mean I
wouldn't say that they're clickbait II
because they are indeed good books and I
don't think that they cross any line but
but you know the that's just a decision
you have to make right like the people
who who write classical recursion theory
by pure OD Freddie he was fine with the
yellow title and the and nothing more
whereas I think other people who who
wrote a more popular type book
understand that they need to have a
compelling cover and a compelling title
and and you know I don't think there's
anything really wrong with that we do we
do take steps to make sure that there is
a line that you don't cross and if you
go too far maybe your thumbnails
especially racy or or you know it's all
cats with too many exclamation points we
observe that users are kind of you know
sometimes offended by that and so so for
the users who were offended by that we
will then depress or suppress those
videos and which reminds me that there's
also another signal where users can say
I don't know if was recently added but I
really enjoy it just saying I don't I
didn't something like I I don't want to
see this video anymore or something like
like this is a like there's certain
videos just cut me the wrong way like
just just jump out at music I don't
wanna I don't want this and it feels
really good to clean that out to be like
I don't that's not that's not for me I
don't know I think that might have been
recently added by this that's also a
really strong signal yes absolutely
right we don't want to make a
recommendation that people are unhappy
with and that makes me that particular
one makes me feel good as a user in
general and as a machine learning person
because I feel like I'm helping the
algorithm my interaction I need you
don't always feel like I'm helping the
algorithm like I'm not reminded of that
fact
like for example Tesla and Otto Pollan
you know on musk create a feeling for
their customers for people their own
test is that there
helping the algorithm of testify like
they're all like a really proud they're
helping nicely learn I think YouTube
doesn't always remind people that you're
helping the algorithm get smarter and
for me I love that idea like we're all
collaboratively like Wikipedia gives
that sense they were all together
creating a beautiful thing
YouTube is uh doesn't always remind me
of that it's uh
this conversation is Right any of that
but well that's a good tip we should
keep that fact in mind when we design
these features well I I'm not sure I I
really thought about it that way but
that's a very interesting perspective
it's an interesting question of
personalization that I feel like when I
click like on a video I'm just improving
my experience it would be great it would
make me personally people are different
but make me feel great if I was helping
also the YouTube algorithm broadly say
something you know saying like there's a
that I don't know if that's human nature
we you want the products you love and I
certainly love YouTube like you want to
help it get smarter and smarter smarter
because there's some kind of coupling
between our lives together being better
if if YouTube was better than I will my
life will be better and that's that kind
of reasoning I'm not sure what that is
and I'm not sure how many people share
that feeling it could be just a machine
learning feeling but at that point how
much personalization is there in terms
of next video recommendations so is it
kind of all really boiling down to a
clustering like you find in ears
clusters to me and so on and that kind
of thing or how much is person s to me
the individual completely it's very very
personalized so your experience will be
quite a bit different from anybody
else's who's watching that same video at
least when they're logged in and the
reason is is that we found that that
users often want two different kinds of
things when they're watching a video
sometimes they want to keep watching
more on that topic or more in that genre
and other times they just
are done and they're ready to move on to
something else and so the question is
well what is this something else and one
of the first things one can imagine is
well maybe something else is the latest
video from some channel to which you've
subscribed and that's gonna be very
different from for you than it is for me
right and and even if it's not something
that you subscribe to it's something
that you watch a lot and again that'll
be very different on a person-by-person
basis and so even the watch next as well
as the homepage of course is quite
personalized so what we met some of the
signals but what a success look like
what a success look like in terms of the
algorithm creating a great long-term
experience for a user or put another way
if you look at the videos I've watched
this month how do you know the algorithm
succeeded for me I think first of all if
you come back and watch more YouTube
then that's one indication that you've
found some value from it so just the
number of hours is a powerful indicator
well I mean not the hours themselves but
the fact that you return on another day
so that's probably the most simple
indicator people don't come back to
things that they don't find value in
right there's a lot of other things that
they could do but like I said I mean
ideally we would like everybody to feel
that YouTube enriches their lives and
that every video they watched is the
best one they've ever watched since
they've started watching YouTube and so
that's why we survey them and ask them
like is this one to five stars and so
our version of success is every time
someone takes that survey they say it's
five stars and if we ask them is this
the best video you've ever seen on
YouTube they say yes every single time
so it's hard to imagine that we would
actually achieve that maybe
asymptotically we would get there but
but that would be what we think success
is it's funny have recently said some
way
I don't know maybe tweeted but that Ray
Dalio has this video on the economic
machine I forget what it's called but
it's a 30-minute video and I said it's
the the greatest video I've ever watched
on YouTube it's it's like I watched the
whole thing and my mind was blown is a
very crisp clean description of how the
at least the American economic system
works
it's a beautiful video and I was just I
wanted to click on something to say this
is the best thing this is the best thing
ever please let me I can't believe I
discovered it I mean the the views and
the likes reflect its quality but I was
almost upset that I haven't found it
earlier and wanted to find other things
like it I don't think I've ever felt
that this is the best video ever and
that was that and to me the ultimate
utopia the best experiences were every
single video where I don't see any of
the videos I regret in every single
video I watch is one that actually helps
me grow helps me enjoy life be happy and
so on well so that's that's that's a
heck of uh the thought that's one of the
most beautiful and ambitious I think
machine learning tasks so when you look
at a society as opposed to any
individual user
do you think of how YouTube is changing
society when you have these millions of
people watching videos growing learning
changing having debates do you have a
sense of yeah what the big impact on
society is because I think it's huge but
you have a sense of what direction we're
taking this world well I mean I think
you know openness has had an impact on
society already there's a lot of what do
you mean by openness well the fact that
unlike other mediums there's not someone
sitting at YouTube who decides before
you can upload your video whether it's
worth having you uploaded
or worth anybody seeing it really right
and so you know there are some creators
who say like I I wouldn't have this
opportunity to
to reach an audience Tyler Oakley often
said that you know he wouldn't have had
this opportunity to reach this audience
if it weren't for YouTube and and so I
think that's one way in which YouTube
has changed Society I know that there
are people that I work with from outside
the United States especially from places
where literacy is low and they think
that YouTube can help in those places
because you don't need to be able to
read and write in order to learn
something important for your life maybe
you know how to do some job or how to
fix something and so that's another way
in which I think YouTube is possibly
changing society so I've worked at
YouTube for eight almost nine years now
and it's fun because I meet people and
you know you tell them where they where
you work you say you work on YouTube and
they immediately say I love you too Yeah
right which is great makes me feel great
but then of course when I ask them well
what is it that you love about YouTube
not one time ever has anybody said that
the search works outstanding or that the
recommendations are great what they
always say
when I ask them what do you love about
YouTube is they immediately start
talking about some channel or some
creator or some topic or some community
that they found on YouTube and that they
just loved yeah and so that has made me
realize that YouTube is really about the
video and connecting the people with the
videos and then everything else kind of
gets out of the way so beyond the video
it's an interesting because you kind of
mentioned creator what about the
connection with just the individual
creators as opposed to just individual
video so like I gave the example of Ray
Dalio video that the video itself is
incredible but there's some people or
just creators that
I love that they're one of the cool
things about people who call themselves
youtubers or whatever is they have a
journey they usually almost all of them
are hurt they suck horribly in the
beginning and then they kind of grow you
know and then there's that genuineness
in their growth so you know YouTube
clearly wants to he