Kind: captions
Language: en
the from the perspective of deep
learning frameworks you work with fast
AI tickler this framework and pi torch
intensive flow what are the strengths of
each platform your perspective so in
terms of what we've done our research on
and taught in our course we started with
Theano
and care us and then we switch to tensor
flow and care us and then we switched to
PI torch and then we switched to PI
torch and fast AI and that that kind of
reflects a growth and development of the
ecosystem of dig learning libraries
siano intensive flow were great but
we're much harder to teach and do
research and development on because they
define what's called a computational
graph upfront less data graph where you
basically have to say here are all the
things that I'm going to eventually do
in my model and then later on you say ok
do those things with this data and you
can't like debug them you can't do them
step-by-step you can't program them
interactively in a Jupiter notebook and
so forth pi torch was not the first 4pi
torch was certainly the there's the
strongest entrant to come along and say
let's not do it that way let's just use
normal Python
and everything you know about in Python
is just going to work and we'll figure
out how to make that run on the GPU as
in when and necessary that turned out to
be a huge a huge leap in terms of what
we could do with our research and what
we could do with our teaching and
because it wasn't limiting yeah I mean
it was critical for us for something
like dawn Bench to be able to rapidly
try things it's just so much harder to
be a researcher and practitioner when
you have to do everything up front and
you can inspect it problem with pay
torch is it's not at all accessible to
newcomers because you have to like write
your own training loop and manage the
gradients and all this stuff and it's
also like not great for researchers
because you're spending your time
dealing with all this boilerplate and
overhead rather than thinking about your
algorithm so we ended up writing this
very multi-layered API that at the top
level you can train a state-of-the-art
neural network in three lines of code
and which kind of talks to an API which
talks from OPI which talks from API
which like you can deep dive into at any
level and get progressively closer to
the Machine kind of levels of control
and this is the first day a library
that's been critical for us and for our
students and for lots of people that
have one big learning competitions with
it and written academic papers with it
it's made a big difference we're still
limited though by Python and
particularly this problem with things
like recurrent neural Nets a where you
just can't change things unless you
accept it going so slowly that it's
impractical so in the latest incarnation
of the course and with some of the
research risked out now starting to do
we're starting to do stuff some stuff in
Swift
I think we're three years away from that
being super practical but I'm in no
hurry I'm very happy to invest the time
to get there but you know with with that
we actually already have a nascent
version of the first AI library for
vision running on switch 19 so far
because a Python for tensorflow is not
going to cut it it's just a disaster
what they did was they tried to
replicate the bits that people were
saying they like about a torch the is
kind of interactive computation but they
didn't actually change their
foundational runtime components so they
kind of added this like syntax sugar
they called TF eager tons of logging in
which makes it look a lot like PI torch
but it's 10 times slower than PI torch
to actually hmm do a step so because
they didn't invest the time and like
retooling the foundations cuz their code
base is so horribly company yeah I think
it's probably very difficult to do that
kind of rejoin yeah well particularly
the way tensorflow was written it's
written by a lot of people very quickly
in a very disorganized way so like when
you actually look in the code as I do
often I'm always just like oh god what
were they thinking it's just it's pretty
awful so I'm really extremely negative
about the potential future if it by the
flaws of the FET Swift for tensorflow
can be a different beast altogether it
can be like it can basically be a layer
on top of M lar that takes advantage of
you know all the great compiler stuff
that Swift builds on with LLVM and yeah
it could be a think it will be that's
what you fantastic well you inspired me
to try
Evan Roo truly felt the pain of
tensorflow 2.0 Python it's fine by me
but yeah but it does the job if you're
using like predefined things that
somebody's already written but if you
actually compare you know like I've had
to do because I've been having to do a
lot of stuff with 10 so far recently you
actually compare like okay I want to
write something from scratch yeah like I
just keep finding is like oh it's
running ten times slower than pi torch
so is the biggest cost let's throw
running time out the window how long it
takes you to program that's not too
different now thanks to chance to flow
eager that's not too different but
because because so many things take so
long
to run yeah you wouldn't run it at ten
times slower like you just go like oh
this is taking so long
yeah and also there's a lot of things
that are just less programmable like TF
data which is the way data processing
works and tensor flow is just this big
mess it's incredibly inefficient and
they kind of had to write it that way
because of the TPU problems they
described earlier so I just you know I
just feel like they've got this huge
technical debt which they're not gonna
solve without starting from scratch so
here's an interesting question and if
there's a new student starting today
what would you recommend they use well I
mean we obviously recommend fast AI and
pi torch because we teach new students
and that's what we teach with so we
would very strongly recommend that
because it will let you get on top of
the concepts much more quickly so then
you'll become an extra and you'll also
learn the actual state-of-the-art
techniques you know so you actually get
world-class results honestly it doesn't
much matter what library you learn
because switching from China to MX net
to tensor flow to PI torch is gonna be a
couple of days work as long as you
understand the foundations well but you
think we'll Swift creep in there as a
thing that people start using not for a
few years particularly because like
Swift has no data science community
libraries Oh basil wing and the Swift
community has a total lack of
appreciation and understanding of
numeric computing so like they keep on
making stupid decisions you know for
years they've just done dumb things
around performance and prioritization
that's clearly changing now because the
developer of chris Christie at developer
of Swift Chris Latner is working at
Google on Swift Potenza flows so like
that's that's a priority it'll be
interesting to see what happens with
Apple because like Apple hasn't shown
any sign of caring about
numeric programming in Swift so I mean
hopefully they'll get off their ass and
start appreciating this because
currently all of their low-level
libraries are not written in Swift
they're not particularly swifty at all
stuff like ml they're really
pretty rubbish so yeah so there's a long
way to go but at least one nice thing is
that Swift for tensorflow can actually
directly use Python code and Python
libraries in a literally the entire
lesson one notebook a first AI runs in
Swift right now in Python mode so that's
that's a nice intermediate thing
you