We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Introduction to Deep Learning

00:00

Formal Metadata

Title
Introduction to Deep Learning
Title of Series
Number of Parts
94
Author
License
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Deep learning has been on a hype peak for the last 2-5 years, and it's being adopted by smaller startups and bigger enterprises alike. As an enthusiastic developer, you might be interested in getting in touch with deep learning to see what’s the hype all about, and if you’re not, you should! Deep learning could be a new way of looking at problems and developing innovative ways of solving them. By the end of this talk, you will have a good understanding of what is out there in the deep learning world, including frameworks, languages, popular existing deep learning networks, cloud providers, and more. This talk will be in three parts, first, we will present the most common deep learning frameworks, their advantages and disadvantages, and what languages they support. Then we will talk about cloud providers that support deep learning. In the last section, we will talk about our experience with deep learning on the cloud. For many frameworks, Python seems to be the preferred language, largely due to its ease of use and extensive community support. However, most frameworks also support other languages like C++, R, and Java. For the most part, a combination of preferred language and the intended use case can determine the options for a deep learning framework. Currently, there are many deep learning frameworks, the most common ones are Tensorflow, Keras, Pytorch, Caffe, and Theano. Some of them excel in image classification challenges while others are more suitable for natural language processing or sentiment analysis. Your choice for the most suitable framework should depend on many factors, including the intended application, preferred language of use, and the availability of good documentation and community support. Usually, frameworks default to running on CPUs and need an extra setup to run on GPUs, but these steps are such a trivial task that it's not an important factor to consider. GPUs have a huge advantage in training and inference speed over CPUs due to their parallelization capabilities. These days there are countless cloud providers for computing power, and many of them have responded to the increase of interest in this category of AI by providing VMs specially tailored for deep learning. All the big players (Google Cloud, AWS, Azure) provide GPU enabled machines with deep learning frameworks pre-configured. However, there are smaller cloud providers who offer interesting models in order to survive the competition. When deciding on which providers to go with, one should consider many factors including pricing model, actual computing power delivered, availability regions, and the existence of pre-configured VMs for deep learning. For example, AWS has a good pricing model and great choices of deep learning frameworks, but for a while, it was impossible to provision a new machine as all of the resources were already booked. It’s also worth mentioning that some cloud providers offer trial periods or sign up credit to test their services, so always be on a look for that! At Viaboxx we are interested in trying out new technologies, so we started with deep learning by creating and running a deep learning showcase on a cloud provider. We used Keras with Tensorflow backend for the source code and Kaggle as a source of data. For a VM to perform the training we compared different cloud providers and chose a smaller one, Paperspace. Although the experience was not hiccups-free, we ended up with a good understanding of the technologies and stacks involved.
15
Thumbnail
1:16:56
20
Thumbnail
59:24
23
Thumbnail
10:02
48
Thumbnail
50:08
56
76
Thumbnail
12:05
FreewareOpen sourceSoftware frameworkComputer networkCloud computingProcess (computing)Price indexSoftware engineeringSubject indexingSoftwareFuzzy logicCuboidDifferent (Kate Ryan album)Artificial neural networkXMLUMLLecture/ConferenceComputer animation
Software frameworkComputer networkCloud computingProcess (computing)Price indexSoftwareCloud computingSoftware frameworkInteractive televisionPhysical systemProcess (computing)Sheaf (mathematics)Computer animation
Software frameworkComputer networkCloud computingProcess (computing)Price indexSoftware frameworkCartesian coordinate systemResultantInternet service providerMobile appCuboidSoftwareComputer animation
Computer networkSoftware developerCuboidNumberSoftware developerProjective planeSoftware industryComputer animation
Artificial intelligenceMachine learningSubsetFeasibility studyMach's principleStatisticsMachine learningSubsetArtificial neural networkMachine learningVirtual machineInteractive televisionWordAlgorithmSoftware frameworkBefehlsprozessorMoore's lawSlide ruleMultilaterationPairwise comparisonGraphics processing unitFilter <Stochastik>Computer architectureParallel portCoefficient of determinationSoftwareBitDiagram
Mathematical modelHierarchyMereologyObject-oriented programmingAbstractionLevel (video gaming)FaktorenanalyseShape (magazine)Coefficient of determinationVirtual machineVisualization (computer graphics)Computer-assisted translationGraph coloringFilter <Stochastik>Different (Kate Ryan album)Task (computing)Roundness (object)Object-oriented programmingMedical imagingCASE <Informatik>Latent heatLine (geometry)Phase transitionMathematical modelDesign by contractComputer animation
InferenceStructural loadMathematical modelComputer networkCoefficient of determinationPhase transitionComputer-assisted translationMathematical modelSoftware frameworkArray data structureProcess (computing)Cartesian coordinate systemFilter <Stochastik>MultiplicationMedical imagingInferenceComputer animation
InferenceStructural loadMathematical modelComputer networkProcess (computing)Coefficient of determinationComputer-assisted translationFilm editingDifferent (Kate Ryan album)Operator (mathematics)InferenceMathematical modelMedical imagingSoftware frameworkComputer fileElement (mathematics)Computer animation
Software frameworkRankingPower (physics)Transport Layer SecurityAreaLibrary (computing)Software frameworkRankingCuboidCartesian coordinate systemDataflowSoftware testingPower (physics)Heat transferAlgorithmBitArithmetic meanComputer animation
Power (physics)Software frameworkRankingMathematical analysisVideoconferencingSeries (mathematics)Visualization (computer graphics)WhiteboardTensorPattern recognitionComputer-generated imageryArithmetic meanPoint (geometry)Slide rulePower (physics)Software frameworkGodPattern recognitionWhiteboardMedical imagingCartesian coordinate systemPlanningTime seriesProcess (computing)VideoconferencingMathematical analysisCurveTensorComputer animation
Process (computing)Library (computing)SequenceFunctional programmingSoftware frameworkComputer-generated imageryReinforcement learningRekursives neuronales NetzInferenceImage processingScale (map)MultimediaMachine visionVideo gameBitProcess (computing)Computing platformDifferent (Kate Ryan album)FacebookImage processingLine (geometry)Medical imagingProgramming languageCognitionSoftware frameworkPairwise comparisonType theoryComputer programmingTouch typingComputer configurationRevision controlMachine visionCartesian coordinate systemComputer-assisted translationFormal grammar10 (number)Artificial neural networkLevel (video gaming)Coefficient of determinationNatural languageGame controllerProduct (business)SoftwareGroup actionCodeDataflowLimit (category theory)Library (computing)Speech synthesisChaos (cosmogony)Recurrence relationWhiteboardInferenceNatural numberLoop (music)MereologyProjective planeMultimediaGoodness of fitXMLUML
outputArtificial neural networkPattern languageFunctional programmingComputer-assisted translationAlgorithmCoefficient of determinationConnected spaceSoftwareArtificial neural networkMathematical modelWeightComputer animation
Computer networkSoftwareBitResultantLoop (music)SubsetLengthMarginal distributionWeightConnected spaceMedical imagingPattern recognitionRevision controlMereologyPopulation densityArtificial neural networkSurface of revolutionResonatorClique-widthComputer animation
GoogolComputer networkWeightRevision controlSoftwareProcess (computing)Combinational logicServer (computing)Vulnerability (computing)Cartesian coordinate systemCloud computingPoint (geometry)InferenceFilter <Stochastik>Software frameworkBitInternet service providerComputer animation
Software frameworkGoogolComputing platformCloud computingService (economics)FrequencyFreewareCloud computingSelectivity (electronic)Software frameworkMedical imagingAxiom of choiceInternet service providerVirtual machineSharewareCartesian coordinate systemComputer animation
Service (economics)FrequencyFreewareSoftware frameworkCloud computingMathematical modelSpacetimeService (economics)Virtual machineAxiom of choiceMathematical modelVirtualizationInternet service providerComputer configurationCuboidSlide ruleCartesian coordinate systemCloud computingShared memoryMachine learningInformation privacyBitPhysical lawComputer animation
Task (computing)Internet service providerCloud computingSource codeMobile WebMetric systemInferenceCuboidCoefficient of determinationSoftware frameworkSpacetimeTask (computing)Internet service providerComputer-assisted translationComputing platformLevel (video gaming)Flow separationComputer animation
Task (computing)Internet service providerCloud computingSource codeMobile WebInferenceMetric systemMetric systemSoftwareSet (mathematics)Validity (statistics)Artificial neural networkPredictabilityComputer architectureExpected valueRevision controlMultiplication signBoolean algebraCuboidPopulation densityMedical imagingMathematicsParameter (computer programming)MereologyPoint (geometry)Computer-assisted translationNumberWeightCombinational logicSoftware developerInsertion lossConvolutionForestTask (computing)Library (computing)Mobile WebMathematical modelMultiplicationInferenceProcess (computing)Different (Kate Ryan album)BijectionArithmetic meanComputer animation
DecimalNumeral (linguistics)Task (computing)Internet service providerCloud computingSource codeMobile WebMetric systemInferenceComputer architectureField (computer science)BitDecision theoryResultantBinary codeCoefficient of determinationComputer-assisted translationMathematicsMereologyRoundness (object)Slide ruleSoftwareTask (computing)Medical imagingCASE <Informatik>Interactive kioskSet (mathematics)Computer animation
DecimalBlock (periodic table)Different (Kate Ryan album)Mathematical modelSoftwareValidity (statistics)CurveInsertion lossNumberInformationWeightResultantComputer-assisted translationElectronic mailing listSlide ruleCoefficient of determinationBijectionPopulation densityComputer animation
Data miningScalable Coherent InterfaceSoftware2 (number)Multiplication signResultantTwitterTheory of relativityRepresentation (politics)Computer animation
Data miningBitBefehlsprozessorGraphics processing unitPairwise comparisonResultantComputer animation
Multiplication signMereologyOperator (mathematics)Medical imagingPhase transitionMathematical modelSet (mathematics)Point (geometry)Artificial neural networkResultantRight anglePulse (signal processing)BefehlsprozessorPerspective (visual)BitInsertion lossWeightElectronic data processingSoftware2 (number)Form (programming)1 (number)Coefficient of determinationComputer-assisted translationCartesian coordinate systemReal numberVideo gameCurvaturePairwise comparisonPlateau's problemSlide rulePower (physics)Different (Kate Ryan album)MultiplicationTraffic reportingValidity (statistics)RandomizationComputer animation
Set (mathematics)Device driverMedical imagingInstance (computer science)WebsiteProjective planeDifferent (Kate Ryan album)Right angleMereologyProcess (computing)Slide ruleCodeNeighbourhood (graph theory)Point (geometry)BitMultiplication signCoefficient of determinationFigurate numberVirtual machineIntegrated development environmentSoftwareComputer-assisted translationGoodness of fitImage resolutionCASE <Informatik>Cartesian coordinate systemDecision theoryInsertion lossFilm editingPredictabilityPower (physics)SpacetimeCombinational logicSoftware frameworkLibrary (computing)ResultantMathematicsRepresentation (politics)Descriptive statisticsEstimatorComputing platformGraph coloringSocial classComputer animation
Mathematical modelProcess (computing)Image processingFormal languageComputer-assisted translationSoftwareProcess (computing)BitWeightFormal languagePoint (geometry)Software frameworkDecision theoryResonatorNatural languageInternet service providerElectronic program guidePattern recognitionAxiom of choiceLimit (category theory)Cartesian coordinate systemArtificial neural networkSpacetimeMobile WebMathematical modelCoefficient of determinationComputer animation
MereologyHeat transferMathematical modelSoftware testingMultiplication signCartesian coordinate systemProjective plane2 (number)CollaborationismSoftwareComputer-assisted translationCoefficient of determinationParameter (computer programming)outputProcess (computing)WordIntegrated development environmentCASE <Informatik>Online helpConfiguration spaceSound effectFormal languageNatural languageVariable (mathematics)Programming languageComputer animationLecture/ConferenceJSON
Open sourceFreewareXMLComputer animation
Transcript: English(auto-generated)
Hello and welcome everyone, my name is Haytham and I've been a software engineer for the last five six years Today I'm gonna talk to you about deep learning. It's just a quick introduction
with the goal of maybe You're interested to know what this is about and you're interested to maybe Try it out. See what's all the fuzz about Yeah, so let's dive into the index So just a minute introduction about who we are fair box the company that I'm working with
Then we will move quickly to a quick introduction about what's AI artificial intelligence What's deep learning and how are they connected and will we will discuss briefly? the difference between how humans learn and how machine learns
Yeah, and we'll also go quickly through the deep learning Process, so how does deep learning work? In the next Let's say section. We will talk about popular deep learning frameworks so this will not only be an interaction on what is deep learning, but we'll also take a look at the
Ecosystem around it. So what frameworks do we have? What are the deep learning networks that we will also take a look at with a quick introduction and Yeah, so also Popular cloud providers where you can start your system and actually start with the first deep learning
application and Yeah to give you an example of all of that. We will also or I will also present our experiment We had a little experiment in via box and yeah We wanted to show you what we did what we went through and what are what kind of results did we come up with?
so in this Experiment we will be going through Frameworks also networks providers. Why did we choose whatever we choose of these options? And for the conclusion, it's gonna be some kind of tips on which what should you do or what?
Questions, should you ask an answer before you start with your first deep learning? application so if you have You have clear answers for these questions, then you're on a good way to Actually start your own deep learning app
So let's get started. So thea box where we're a company founded in 2009 we located in clinics winter and We're currently 13 employees and looking forward to increase that number so if you're interested we're hiring and
yeah, so we're very active in the developer meetups and bone and cologne and Recently, we're ISO 9001 certified which is a big deal for a software company yeah, we we've recently got the award of innovative through research
Yeah, so as you can see we're very interested in research. We are always and looking out for research projects So what is deep learning so? Deep learning is a subset of a subset of artificial intelligence so for artificial intelligence
it's just a way for machines to mimic human behaviors, that's the whole idea and Machine learning is a subset of that where it uses statistical methods to actually improve the Machine's experience so to get it closer to
How humans let's say interact? this was this is where deep learning comes which is also a subset from machine learning and The whole or the big idea is it's making Deeper layers of training or of
Multilayer neural networks feasible so before it wasn't feasible to go deeper in the neural networks Which we will explain a little bit later Because we didn't have the computer Powers let's say that brings us to the challenges that we had before that we overcome now And that's why we have deep learning which are the availability of data, so
before you didn't have much data now data is all all around you and We also didn't have much computer power so for deep learning. You're doing a lot of parallelization and CPUs are kind of slow in parallelization compared to GPUs
Which if you want to do deep learning correctly you're gonna do it on a GPU machine because it's much much faster and yeah, these are some of the Challenges one more challenge is Frameworks so before if you want to do some kind of deep learning
You're gonna be writing everything on your own, so you're gonna be writing the algorithms You're gonna be writing the filters and the layers and the architecture and everything and how to import data how to Visualize it everything so that's that's really complex to do on your own But fortunately now we have a lot of frameworks which we will cover and in a couple of slides later
So that's deep learning or introduction and now let's compare how? Humans learn or how machine is supposed to learn So let's take dogs for example
so for humor ever since you're a kid you see a lot of dogs a lot of shapes sizes and colors and In your mind you You come up with a visualization that this is how a dog should look like so when you see a new dog that you've never Seen before you have this Visualization and you kind of know that this thing is a dog or a cat or something and
an effort to kind of Do the same in machines? that's how The machine would learn is you have different layers of filters and these filters are Just doing one specific task for example filter a is detecting vertical vertical lines in an image
Another filter is detecting. Let's say edges or round objects or even different in construct contracts, so All these filters combined will actually formulate
A model that represents something in this case It's representing faces, but it can also represent a dog or a cat or whatever object You're trying to learn and model yeah, so and To enable the machine to do this you have to go through a couple of let's say
Phases These phases are the training and inference so this is your deep learning application This is what it's supposed to do first you have to train it which is collect a lot of data
Not just any data you have to prepare it also for example Let's take cats versus dogs. You have to have multiple pictures of cats multiple pictures of dogs and you have to have them labeled that these are dogs and these are cats and You have to do some pre-processing before you start consuming this data, so resize it to a specific size
Maybe flatten it the image, so it's not a two-dimensional array Let's say it's one flat image or one flat array that will make it easier to work with for the machine yeah, then you can actually start your training process the
The framework or the engine will consume all this data and try to apply these filters and layers and come up with the model which is your Target here you want a model that represents cats or dogs, and it's able to come up or Identify the difference between them
So this model is usually saved as a file and it can be used in the next operation which is inference so for inference you're actually loading this model that you just created and you're Now ready to classify new images of cats and dogs
You just send a request to this Model and you ask it is this picture of a cat or is this picture of a dog? And it will be able to tell the difference with the with within an accuracy element of accuracy, so It's really hard to find something that's 100% accurate
But the more data you have the better or the closer you are to this 100% but you're probably never going to get there because nothing is that accurate would say yeah, so to do this these Two processes you don't have to write everything on your own as we said you have frameworks
And these frameworks Are getting let's say more popular and These frameworks are it's just a neat library or tool that enables you to
start deep learning application without diving into the algorithms and the Really small details that you don't really want to care about so and yeah, these are some of the Frameworks that are out there, and they're right by Statista Which on a based on a power ranking as we can see tensorflow is
The really the most popular or powerful one so just an background on why or how did they rank them? They're looking at Google search, so Google searches GitHub activity
and Yeah many other Yeah articles box Anything publications around these frameworks, and they're basing the score on that as we can see TensorFlow and Keras are really the top two and TensorFlow is way ahead of the others
Now we will be talking a little bit about advantages its advantages of the first four frameworks Up until now any questions Yeah, please
test Yeah, my question is It says power ranking, but what does it mean?
It's like the the ability of the frameworks or like the the co-abilities or what what does it power mean in that regard? So the power is not Not directly related to the capabilities of the framework or let's say the performance but on the popularity that
Let's say as we say it from How many people are using it on how popular is it and how? Easy is it to? To find let's say documentation for this thing So that's why tensorflow is really on the top as we can see This is a good point to switch to the next slide. Why is
Sorry, why is tensorflow is so predominant so just a little background TensorFlow was developed by Google brain team and it was intended to be only for internal use they At the first they didn't plan on often sourcing it, but they did and
Ever since it's the most popular Framework and it has really good documentation both at the official and community support and It has this really powerful tool tensor board, which will help you visualizing
debugging or even Getting get in more details on the deep learning process. So it's not just that it's good it's also well documented and it's really being developed and worked on and One also big advantage is you can use it using Python C++ and
Usually the applications are text based or image based recognition classification Yeah, as you can see also sounding sound recognition and time series analysis video analysis all these Applications let's say and it has this one
Disadvantage, which is a steep learning curve. So if you're starting with tensorflow You're gonna face some challenges at the first steps to actually be proficient in it, so this is one point to consider if you're choosing to start work with this and
this this problem was solved by Keras, which is Developed by a guy that used to work in Google and he thought tensorflow is a little bit Too low-level, let's say too complex. So he came up with Keras, which is a high-level library it makes your life much much easier and
you can just start and just by reading the lines, you know what they're kind of doing and Yeah, so it's not an independent by itself. It can work on top of tensorflow and Tiano and CNTK which is from Microsoft Microsoft cognitive tools
Yeah, so you have some options, but if you if we look down here It's has support only for Python. So you're limited to using Python if you don't know Python You're gonna have to learn Python a little bit Yeah, and it supports sequential and functional styles, so depends on your
Type of programming and As it's a high-level library, you kind of have less control For these details, for example, you can still use tensorboard to view your learning process But you cannot in the code be as more specific in details as you can with tensorflow for example
So yeah, these are the two most prominent deep learning frameworks so far But we have a very good contender here, which is PyTorch PyTorch is really picking up now lately for the last year or so
It's developed by the research group in Facebook Yeah Yeah, so and yeah, so it's easy to use much easier to use. Let's say them TensorFlow because it's Pythonic in nature. So if you know Python you can easily just program and
It's really good in training and classification. It's a little bit faster than tensorflow for to be to do a comparison yeah, and Mostly the applications are image detection and classification natural language processing and
Re-inform reinforcement reinforcement learning Yeah going back also to the disadvantages only Python and It's not as mature as tensorflow or Keras So it's only produced or released for production in December last year
So people are not really comfortable to go with it to production level applications just yet And now the fourth one From top was cafe Cafe was developed first as a research project in the University of California, Berkeley and
It's written in C. That's why it's really quick with working with images as we can as we will tell and interesting about thing about cafe that yahoo kind of Took cafe and integrated it with spark and to create cafe on spark which is distributed deep learning
Let's say platform our framework and also Facebook Developed cafe to they took cafe developed cafe too and then integrated it into Pi torch, which we saw in the previous example Yeah, and big advantages are speed of image processing
During the learning process Which is consuming the data and during the inference when you're trying to classify Docs versus cats for example, and you can use it with many different languages CC plus Python MATLAB, so you have more options there
Currently it's mostly used by For academic research, but it also some have some large-scale applications vision speak speech and multimedia Coming down to the Limitations so we don't have support for recurrent neural networks, which are special
or special part of neural networks where kind of the training the the say the layers are connected in a way that can represent the loop and Yeah, it's a little bit harder to get started with to learn than say the Python based
Frameworks or yeah For now any questions Right Yeah
No, no, okay Take a look into that. Yeah So the mark was that Keras is also available in our so we I need to do a little bit more research on that
Thank you Yeah, one one small note also that Keras will be integrated in the new version of tensorflow. So they're currently working on tensorflow 2.0 Which will have Keras built-in inside. So
Yeah, that's a good step. You're having instead of using two frameworks together. You're only using tensorflow and has these capabilities Yeah, so we've been talking about Neural networks or What are they are worried? Why are they named like this? So a neural network is something or a bunch of algorithms that are
Loosely Modeled after the human brain so we can see kind of the connection where we have these neurons or nodes and they're Interconnected with each other. That's where the Name came up. It's it's not because of a functionality on how they or how they work, but it's loosely on how they look like and the
yeah, the the Say the purpose of these layers is to detect patterns So you detect smaller patterns and combined combine all these patterns together Let's say and you end up with a model for something bigger cat or dog or bicycle or a desk or anything
yeah, and now we'll take a look at a couple of the the most popular deep learning networks The first one which was deep let's say it was only eight layers deep it was called the Alex net and
It was a revolution because no one ever before built Neural network that is eight layers deep that was in 2012 and it won a Contest by a really large margin over the other competitors. So it's a classification
Contest that's happening yearly and They just took the first place and now the world kind of realized that if you have more layers, then you have more More accuracy you have better results
Yeah Next next Let's say network that we want to talk about is the residual networks Resonate it has multiple or multiple versions, but the idea of this network is we have this loop or this
Skipping part where a layer is a skipping a layer So you have a connection skipping a layer and that's done to Have more data in the next layers. So just a quick or for each layer when you're doing the Pattern recognition you're using you're losing a little bit of data from the the original image
So if you start with a really large image about five layers Let's say later you have a subset of the image or much dense image and it's it's lost some data So with this technique you can have more
More width more length in the image or more data and then you can actually stack up more layers So instead of eight We have now 50 or even more Yeah so resonance enable that to go much much deeper and
That stick takes us to Google. Let's which are the inception nets So now we have the first technique which is skipping a layer And now Google came up with a go with the Google net or inception Which has also many versions Instead of having one let's say filter per layer one filter with size
let's say three by three you can have two or three or even four filters as you can see the layers or the The layers are getting Wider not not only deeper So with these techniques combined we're now able to go to hundreds of layers deep
Which was really unprecedented five years from when this came out, for example Yeah, and they kept working on this. We have now inception version 3 which came out 2015 it's also really big really a really deep network and
They started testing with combinations. For example inception with ResNet and came out with inception ResNet, for example Yeah, you can just Google search a little bit more on what other networks are there each network kind of have its weak points or
Strong points and you can just read about each one to know which applications are more useful for this Which which let's say deep CNN is more useful for my application Yeah So Say for example, you know, which network you want to use, you know, which framework you want to use
But now is the question is do I use my machine to set up everything or do I need a cloud? server or cloud service to have this To have my application on the answer is you can have both you can
Set up your machine to do the processing to do the all the two steps let's say the inference and the training but you can also just go on the cloud and it might make more sense for the Let's say for the coming reasons. So one of these providers is Google Cloud and
There you can easily set up machines, but just by clicking next next next selecting couple of stuff Then you have a machine that has let's say TensorFlow 1.13 and it's pretty much configured and ready to use you don't have to do any kind of setup as we can as we will see that many of
Cloud providers already have these kind of machines that are ready to be used The good thing also about Google is you will get a 12-month $300 trial so You can use it for deep learning or other stuff also
Next provider is Amazon Web Services, of course Yeah, they also have a lot of AMI's which are Amazon machine images. They are open to VMs also pre-configured with any choice of frameworks that you want
Yeah, you also get a free trial for 12 months But it's very limited and you probably cannot get a really high-end GPU machine to Do your To do your work or application Yeah There are a lot of other big names
That you can go with but also keep an eye on these smaller Providers for example paper space So when we did our experiment that we will talk about in a bit We actually went with paper space Our first choice what was Amazon, but it was really difficult to get a machine from Amazon even the paid one
They were really booked and there was no way to get to provision a new machine So we started to look somewhere else and we found paper space They have also the machine learning in a box VMS So pre-configured pre-installed you have a GPU machine a real GPU and not just let's say a share of
Virtual GPU let's say and yeah, you can have the pricing model that you want and We found out that they have really excellent customer support Services and yeah, that's why these were the three reasons that we went with paper space say instead of Google or with
With Amazon Yeah, so also worth noting that There are probably a lot more Microsoft Floyd hub Google collab GPU eater even Alibaba has now these kind of machines that will
You can do deep learning on Yeah, so depends on your needs for example With the data protection laws you cannot have data in from the
European Union save somewhere else so that might limit your choices of which providers. Do you want to go with? so just keep these things in mind and you will be or You'll find some options to for cloud for providers Before we jump to our next cloud next
Slide which is about our personal or our experience and fear box anymore questions comments I Don't see okay so at via box So at first we wanted to get
Say to work with fear with the deep learning and We figured okay. We need a goal. We need to choose. Where do we do our stuff? We need to choose which? Providers we're gonna use which Frameworks and our setup was as follows so the task was we want to classify or write a map that
classifies cats and dogs so you You want it you want it at the end to be able to? separate or classify cat or a dog and we went with pepper space for reasons we mentioned before and
We used Kaggle Kaggle is kind of a platform Where you have a lot of data a lot of competitions, it's really getting popular popular lately and yeah, it's If you want to start with deep learning you're gonna hear about Kaggle a lot
There's they do annual or more than annual competitions. Let's say with actually financial rewards for example the cats versus dogs Dataset came from a competition that You can enter subscribe and do your say up and you compete with other people to
Let's say in a specific task and the winner gets this this much these rewards yeah, so we got our data from there and We decided to go with Keras with tensorflow because it's the easiest combination you have the powerful
Tensorflow back back end with Keras as a let's say as a Library to work with which is much much easier, and we didn't need to go into details to change the smaller details of deep learning process, so Keras was enough and was actually
quicker for development and As a part of the experiment we wanted to compare the The performance of multiple CNN's or convolutional neural networks, so we have dense nets and
We have inception that we saw before we have the ResNet also We have the inception ResNet v2 and we have a couple of mobile nets Which are really smaller nets that are designed to be quick efficient, and they can actually Eventually run on anything they're that compact and that slow that small
Yeah, and we had We kept our eye on a couple of metrics accuracy so for each Network after you train. There's a validation or validation step inside the training and it will
Yeah, tell you how accurate your training is yeah Yeah, so there are set for these names are there a set of parameters, or are they just names
So for the dense nets for example. It's the one-to-one is the number of layers 169 is a bigger network and 201 is 201 layers deep network for this for the Inception for example. It's just the version of that
Yeah, mobile net is also the same the version of the network that they came off with so developers will Will develop this network with the with this kind of architecture with this many layers here that many layers there and They kind of give it names based on the architecture itself, and let's say version 1 version 2 these kind of stuff
Yeah Yeah going back to the metrics so we have Accuracy which is how accurate your predictions are for example if you predict 100 images of cats and you only predict correctly 90 of them, then you're at 0.9%
loss is Loss is a tricky thing so it's it's a number that you want to go to zero if you have a higher loss It's that means your network is not It's not behaving correctly, and you need to change some stuff so
the difference between the Reality and the expectation is too high Yeah, and for the training speed so usually The training process takes a lot of time and especially if you're doing it on your local machine, which doesn't have a GPU
One one of these could take up to two three hours to train on Only 2,000 images or so if you want to go for maybe five a box Which we will talk about maybe later, so the training process is relevant because you want to train quickly
You want to be able to adapt to changes in your data set really quickly So training speed is relevant here, and we also have the inference speed where if I give this Model or this network an image. How fast will I get an answer? so we kept our eye on these kind of
metrics and Questions again, yeah So yeah
Yeah, so the question was For your problem, which how do you decide which? Network to go with that's an excellent question. Thank you so usually Each as we said each one has its own let's say Its own field that it excels in but all of them for example can do cats versus dogs
And this is was this was part of our experiment Let's say which one let's see which one is better as in the binary classification for example task but you can also just Depending on your your use case you have to search around see what's what's the best?
network there and It comes with a little bit of experience Maybe actually none of them is 100% fitting your architecture or your purpose And you need to actually go into these networks and check tweak a little bit change a little bit of details
That can be done for example using Keras where you can go to specific layers out of these 201 layers and just do changes here and there so as you get more experienced you will get to Know which which network is Suitable for this task or not, but for image classifications
Most of these or all of these are good enough. Let's say as we will see in the next slide More questions Yeah, so We did our experiment we set up our machine we
Push the data in we trained each network for 10 epochs Which is 10 rounds of training on the data set? Yeah, we came up with these results it might not be that much clear, but the goal or the
the result of this So we're looking at the validation accuracy how accurate these networks are and Deciding if this is a cat or this is dog as we can see on the beginning. There's a well There was a big difference the lowest two curves that you see are for the mobile nets which are not really deep so
You have less information. They're less less of a model to to to do the Classification and you can see that the deeper networks like dense net 201 169 121 or even the inception
Inception resnet yeah, these are the most Or the largest networks, and they ended up with the highest validation accuracy So that's a suggestion that the deeper the network it usually means It's you're gonna get more accurate results
Yeah, but so We don't have a slide for this, but talking about loss because we wanted to keep an eye on that These also deeper networks have lesser loss so the dense nets all had the same loss, which is was a really small number and
All the other bigger networks kind of had also smaller loss and Looking at the mobile nets which are really small they had higher loss than the rest And now we can look at the relation between the depth and the
training duration so we run for 10 epochs and so 10 10 times the training on the same or on the data and As you can see dense net took about 700 something Seconds that's in seconds, so
Forgot how much that in minutes 12 minutes to run 10 epochs Which is not bad because you're pushing so much data, and you're getting 90 something 97 percent accuracy results The you can see the trend here is the deeper network it is it's gonna be takes so much time to train and
this is also representative of the Classification time so also the deeper the network the more time. It's gonna need to classify and Yeah, so this training was done on a GPU. I think it was
K 8000 from Nvidia, so it's a really powerful machine, and it was it needed 12 minutes for example for the Dense net 201 if you want to do the same training on this Mac Let's say might take up to four hours or something to do the same or maybe more
I think even maybe more just to do the same set so this gives you an idea of How much GPUs are better than CPUs in doing the training? Which you always need to keep in mind do I really need the? Training to be that quick or am I okay with waiting a little bit for
Reducing the cost and say of the GPU, but also I have to wait a little bit more so these are comparisons that you have to do Yeah, so giving these results. We kind of came up with some conclusions
We like the dense net the most Because it had a really high accuracy really low loss and Yeah, it didn't take that much time to train compared to the other the other bigger two networks, and it had the same accuracy the same loss and
Yeah But it was a little bit slow in the classification time so when you send an image is this a cat or this dog? It's taking a little bit more time to get the reply back For example if you compare it with the mobile nets which are really?
Quick and reply, but they're not that accurate so again you have to do the comparison here Looking at inception, which is one of kind of the older ones It's really quick in classification time because it's not that deep, but the accuracy was not impressive, so yeah even though
This makes more sense From the let's say the classification time perspective, but accuracy is not good enough to actually consider it Yeah, so that was our experience so far Yeah, any questions here about the results. Yeah, please
Yeah So how much time does the classification usually takes? The answer is in milliseconds, so usually I think the
The longest classification time was around 45 milliseconds and also depends on which machine Are you running this classification on so there might be a 10 times slower to run the classification on? this Mac for example Dana Cloud machine which is really powerful, so
depends on your Set up and it depends on how deep the network is Yeah, if your classification is taking let's say one second. That's not a good thing you probably need to improve on Maybe choose a Smaller network or maybe if that's fine by you one second is okay, but in real life applications you want to go to
milliseconds Perspective let's say More questions yeah again, sorry
Okay, so after the training part is done
This same model that you came up with is not increasing because you're not training it again, so you can have Let's say two separate processes where? the training is happening live and You're always have newer the more data or when you put more data it will produce a new model, but
usually the two operations are separate, but you can have a network that's live that is taking data and classifying at the same time it's Also training on this new set of data But usually it makes sense to separate these two because the training is taking so much
CPU power and maybe you're learning the classification up on a smaller device or smaller maybe a mobile or a Set like this so yeah More questions Yeah, and the training it looked like that some
Neural networks are learning like for the first report. They are like over 95 percent right 89 would it be possible to just do two reports of training because it's much less Training time and so we use a deeper neural network and its advantages in the form of less training time because of
That's a valid question too, so If you remember our slide the first two or three of books is where you see Significant improvements, but then it most of them or all of them kind of flat out to this plateau
Let's say and yeah, it makes sense so now you know that if I train to 20 epochs It's not gonna make more difference than training for five So why should I go for 20 which is a valid question, and that's a good point. That's something you should do So am I am I good enough with five if the answer is yes
Then you stop at five because you don't see more improvements going on there Thanks Yeah Okay
So yeah training the data with the training the network with the same data multiple times so you can have something in the pre-processing Preparing the data phase where you don't just take the image at just
Images as they are because that might not be very useful you kind of do some image manipulations where you rotate in In a Random amount and you kind of horizontal, so it's a vertical rotating and kind of changing the perspective It's maybe blurring a little bit is maybe cutting some edges so each time you'll have more data
So you have the basic that's at the same, but from the preparation of data process. You'll have more data. You'll have More or less changed data. That's why from a smaller set of data. You can actually get Really good results or relatively small when I say relatively small two thousand is barely enough
But if you have two million, let's say images, then you're getting better and better results more questions Yeah, please
Okay Okay, so the question is where do you get your data set from and how you how do you decide that these are? Good images are or not so usually What we did and what we recommend is looking at Kaggle
which is the platform that we talked about it has a lot of data on a lot of different stuff and Yeah, you can choose your data set from there. There's always a description of what is this data set representing What's for example? It was images. What are the images sizes
We say average sizes are they Colored images non colored are they let's say recent or not recent So you'll get a lot of details there or you can do go the hard way which is collecting the data yourself You can run around in the neighborhood taking pictures of dogs and cats, but probably you don't really need to do that
yeah, and it's gonna be your decision to see if this is is this data representative of the The process or the goal that I want to to do Yeah, please Did you compare your results to the Kaggle results? So because they provide the solution right they have this
Correct, and I don't think so. It wasn't part. It wasn't in our interest to let's say compete or to Compare with this. We just wanted to do our own
Experiment and see which of these Networks are the best in this case, but yeah, that's a good point I'm now interested to go back and look how what for example the winner Of that competition how much accuracy loss and these values that they have more questions
Yeah Okay Yeah, so the question is How much time and how much difficult is it to set up the whole thing? So code wise let's say depends on how
efficient are you with coding but with Keras you're probably gonna find a lot of examples or a lot of Yeah, a lot of material that will help you build up your application quickly and Regarding the setup with the environment that we did the trainings on so as we said per space has
Machines that are already configured so you don't really need to set up the frameworks and the GPU libraries or the Drivers and this kind of stuff you don't have to do this but unfortunately we kind of wasted or we had to do some changes there because we wanted to update the
Nvidia drivers CUDA drivers, I think they're called and there was a little bit of Incompatibility with these newer drivers versus the The machine or the GPU itself so that took some time, but usually if you just want to run something
And you have the code ready. It doesn't take you more than one minute to actually just run this Python code And it's it starts Doing the class of the training for example but And unknown circumstances can a little bit prolong this so I don't think we took more than
One two working days to get through all these figure out the problems first And then set up the environment the way we want it more questions
Okay, the question is difference between accuracy and loss So accuracy is just looking at how many pictures did you guess 100 pictures? How many did you guess correctly? The answer is it's a 90 then you have a nine point nine accuracy loss looks at
For each image you have a let's say this is a 90% Cat that's what my network kind of figures out then you have this 10% and you have this 15% from the other image and you have let's say another 20% of
Inaccuracy in each classification of each image The amount or the kind of the combination of these is the loss is for each prediction How far are you from the truth? So it's it's a really tricky concept. I would really encourage you to look for someone online That's cat that can explain it a little bit better than I do
But yeah, that's the general idea. How far are you from the the accurate or the correct? Estimation more questions Yeah
Yeah, so question is do you have any experience with customers regarding these? These deep learning projects. The answer is yes, we kind of integrated some of these Techniques or deep learning in general in some research projects that are going on currently
Nothing on let's say customer sites, but more of research projects currently questions okay, moving on to the last slide
Which is if you want to start your deep learning Application what are the things that you should be looking for? So what purpose are you trying to do? Are you trying to do natural language processing or are you trying to do? Cats versus dogs, for example, or are you trying to do any other voice?
Recognition stuff so What is your purpose and that can? Affect the decisions that you're gonna take later Yeah, so that can affect which framework do you want to use?
TensorFlow or pi or pi torch or Keras or yeah, which Which framework is better for you? What language are you more comfortable with this can also affect the frameworks that you're going to choose? so a lot of People prefer Python to C and a lot of academics. Let's say would prefer C C++ to Python
So this is something you need to consider Also Can you use existing popular models? So for example, we were able to use all these networks the resonance the Inception the mobile nets for our purpose everything works. So maybe you have another purpose
You'll have to do a little bit of research. You have to ask around. What's the best network to use for this purpose and The last consideration point
Restrictions or requirements for the providers So we talked a little bit about you cannot have data from the European Union stored outside So that limits your chance or your choices of which cloud providers are you gonna? use
Yeah So these are one or four more important points that you Need to consider and After that, you'll be able to decide What am I gonna do? I'm gonna use this framework. I'm gonna implement this neural network and I'm gonna use it on my machine for example or
Amazon or paper space and yeah That should give you the guide more questions Yeah For your tests, did you did you actually train the full model or just part of it as in?
Transfer learning. Yeah, mostly we trained I Think we froze two layers only and we kind of trained the whole net for that. So it's yeah, that's another technique of Deep learning where you if you have a really big model and you don't want to spend so much time training you can freeze
Let's say the first ten layers or so. So you're not training these layers you're just training the rest that can significantly reduce the Training time and it might have a effect on accuracy, but it shouldn't be that much but in our case we
Most likely or we froze a couple of layers, but we trained the rest Just note here that tomorrow I'll be also doing a workshop where we will be doing the cats versus dogs. So if anyone is interested
we're gonna be doing a workshop on your first deep learning application and Yeah, just run it on your local machine and see how it goes back to more questions Yeah
So team of 13 members, how do you the collaboration between yourself? so That's a good question. So it takes a lot of Initiative a lot of feeling of ownership, let's say to take tasks and see them through
I think this question is more about the environment of the work more than How did we manage to pull through this? but yeah initiative be helpful and Usually One guy is not only focused on one project one small part of a project
You're you're always trying to look beyond your small scope and actually look on What's everyone doing and try to help be involved in more projects? more questions Yeah
Just wondering about Language processing. How do you enter variable lengths? Input into a neural network, I mean some people speak like 30-seconds home like five minutes and how do you put it inside as input? Yeah, so
My experience with natural language processing is not that much but I have had a colleague before who did theses on this and as long as Usually you're working with text and you can configure these parameters or How long do I need sentence to be how many characters or how many?
Yeah, how many words do I need to be in this world So if you have more questions, feel free to join me here after the talk But as we can see it's time is over. Thank you for so much for listening