We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Deploy your Machine Learning Bots like a boss with CI/CD

00:00

Formal Metadata

Title
Deploy your Machine Learning Bots like a boss with CI/CD
Subtitle
Using Gitlab Open Source tools to automate NLP models deployment
Title of Series
Number of Parts
130
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Context: Today is relatively easy to create and train a conversational agent using Machine Learning Techniques, fire it up and showcase it in your computer Problem: Sharing your chatbot with the outside world is not as easy as training your models. Load Balancer, Unit Test, Integration Tests, Differential Tests ... Text Analytics and retrain the models to better serve your audience goes way beyond the simple agent that runs in the developer environment Solution: I want to show how from my experience of deploying bots to production, leveraging DevOps + DataScience skills along with an entry level knowledge of Databases, CI/CD and distributed systems you can take your prototypes to a next level, deploy, iterate and re-train your models faster. Pre-reqs: Entry level understanding of CI/CD Pipelines, NLP, jupyterhub, Version Control, Rasa
61
Thumbnail
26:38
95
106
Machine learningBoss CorporationVirtual machineRobotBoss CorporationGoodness of fitExpert systemTime zoneTouchscreenComputer virusComputer animationMeeting/Interview
Boss CorporationMusical ensembleVirtual machineContinuous integrationChatterbotLibrary (computing)Presentation of a groupE-learningEndliche ModelltheorieComputer fileHand fanAnalytic continuationMusical ensembleCartesian coordinate systemCASE <Informatik>RobotSoftware frameworkGame theoryPersonal digital assistantCurveStudent's t-testComputer animation
Virtual machineLaptopDatabaseEndliche ModelltheorieData storage deviceSoftwareService (economics)PrototypeComputing platformQuicksortSequelComputer animation
View (database)Video gameProcess (computing)MathematicsCASE <Informatik>Integrated development environmentCurveSoftware frameworkMathematical analysisLine (geometry)Dependent and independent variablesChatterbotEndliche ModelltheorieWave packetTunisRobotParameter (computer programming)Data conversionHypercubeCombinational logicProcess (computing)Library (computing)WordLevel (video gaming)DiagramConnectivity (graph theory)Operator (mathematics)File formatProduct (business)Machine codeCycle (graph theory)Multiplication signMereologyPrototypeDataflowVisualization (computer graphics)IterationValidity (statistics)1 (number)Software testingView (database)Type theoryStandard deviationMessage passingProfil (magazine)Cross-validation (statistics)Revision controlSoftware developerVirtual machinePiMusical ensembleExpert systemProcess modelingComputer scienceService (economics)Video gameSoftware engineeringReading (process)Computer animation
View (database)Data conversionEndliche ModelltheorieFeedbackLoop (music)LogicMetric systemThresholding (image processing)Row (database)Software frameworkPoint cloudSoftware testingReal numberCASE <Informatik>Machine learningPoint (geometry)RobotDemo (music)Software developerCross-validation (statistics)WordLibrary (computing)QuicksortIntegrated development environmentFile formatRepository (publishing)MathematicsIterationVirtual machineGoodness of fitThread (computing)Machine codeKnowledge baseServer (computing)Type theoryRepresentational state transferBranch (computer science)Level (video gaming)Product (business)Operator (mathematics)Key (cryptography)Software engineeringChatterbotRevision controlWave packetMereologyView (database)Traffic reportingTrailEnterprise architecturePredictabilityParameter (computer programming)RoboticsVideo gameMoment (mathematics)GoogolComputer animation
Continuous functionContinuous integrationIntelFAQMathematicsRobotCASE <Informatik>Integrated development environmentAuditory maskingVideoconferencingMusical ensembleCommitment schemeContinuous integrationYouTubeSoftware developerLink (knot theory)Pie chartFile formatWordBranch (computer science)Video gameGame theoryWhiteboardMedical imagingLocal ringMultilaterationComputer animation
Server (computing)Asynchronous Transfer ModeTraffic reportingWikiMathematicsTrailProduct (business)Endliche ModelltheorieRobotCommitment schemeDifferent (Kate Ryan album)1 (number)HistogramWave packetLoop (music)QuicksortConfidence intervalPredictabilityComputer fileDistribution (mathematics)Data conversionVirtual machineFeedbackSoftware developerWeb pageLink (knot theory)Server (computing)Branch (computer science)Goodness of fitLevel (video gaming)View (database)VideoconferencingMoment (mathematics)Template (C++)File formatDiagramSoftware testingComputer animation
WordConfidence intervalNumberToken ringType theoryData conversionMereologyLimit (category theory)RobotEvent horizonWhiteboardCASE <Informatik>Computer animation
Continuous integrationCollaborationismTelecommunicationVideo gameComputer animation
PiScripting languageProcess (computing)Point (geometry)Data conversionSoftware developerComputer animation
Open sourceProjective planeLink (knot theory)VideoconferencingMathematicsDemo (music)Server (computing)Point (geometry)
CurveIntegrated development environmentMobile appView (database)Meeting/Interview
Video gameView (database)Cartesian coordinate systemProjective planePoint (geometry)Repository (publishing)Software developerHand fanCore dump
Data conversionDifferent (Kate Ryan album)Group actionWeb pageMultiplication signOnline chatProduct (business)Link (knot theory)GoogolMeeting/Interview
Transcript: English(auto-generated)
So next is William Arias, and he's going to talk about deploying the machine learning bots like a boss using CI CD William Arias you you work for GitLab, is that correct? Hello. Yes, I work at GitLab. Yeah, perfect
So you're the expert No, so right where you joining from by the way, and I am from Colombia But I live in Prague in Czech Republic. All right, so it's a good time zone for you now Okay, excellent, okay, so please start sharing your screen and then you can take it away
Okay. Hello everyone. So my name is William And today I want to share with you some of the learnings I had since I started like back in 2016 working with developing bots and I'm big fan of NLP and the well the use cases of a chatbot and
In this in this talk I want to introduce you to How you can step up your game using principles of continuous integration and continuous delivery using Rasa the framework for chatbots and GitLab CI CD
so why why why I decided that I wanted to share these these learnings with you because When you start working on this as a hobbyist or as professional I have done both Usually what you do is that you start behaving like a one-person band so I was reviewing the agenda of the talks today and
If you have been following as well You can see that there are presentations that they are teaching you how to create a docker file how to deploy a model using Docker how to serve it. So you have to create an API and Besides you have to use other tools For to create a model itself and at the end we start behaving more or less like this
Like like, I don't know if you if you like the symptoms when I was doing that I remember this guy because we are a one one-person band. We are playing all the instruments. So You start becoming or you are expected to deliver an application that contains that has machine learning embedded you more or less have to behave like a one-person band and know a lot of
instruments and to Learn how to play these instruments. There is a learning curve that might be frustrated I was the other day Teaching assistant Yeah teaching assistant in some online course where the students
They were able to create the model for a chatbot but they when they were asked to please deploy to AWS and show us give us an endpoint where we can connect a channel and usable Many of them they start struggling because that's not what you are supposed to necessarily
Be aware of you know, the models, you know How to try how to use different libraries to create the to classify intents to classify entities But not necessarily all the tooling that you are expected to more or less Have some some some knowledge to be able to serve a model so
example Let's say you use tensorflow for something for To create some sort of some classification problem to solve some classification problem so you prototype in Jupyter notebook and once you prototype in Jupyter notebook you If you want to store and probably you will need a database So you need to know which one should I use should I use something like asandra?
for example poser my sequel and then as same before Then if you start digging into this you will For sure, you will find okay So I should deploy it as a container and then qern it is and how do I deploy should I create a virtual machine? Or should I go directly to the platform as a service or should I just use an API software as a service? Whatever?
And next to serve to the other world. So all of these things They are more or less what they call today the fully stock machine learning engineer or data scientist. I don't know and I Want to share with you What was the approach that I follow and how today if your purpose or if the challenge that you are facing is I?
Want to create I don't know my own startup in which I will build chatbots for whatever business and You want to be fast and you want to deploy in a very very like close to? standard professional way but also
With iteration in mind and being fast. So I want to show you how I did it How am I doing it? And this is not like written in stone This can change and all these port flows you if you know the principles then you can adapt it to your own workflow So how is that the how I started to?
Divide this this problem when I say it's taking me too much to deploy a chatbot to production It works in my machine the typical one it works in a machine is very easy to run it in localhost But I want to deploy it and and if you are doing this for yourself as a one person band or for a company You will find these these challenges
So first I started thinking of it like what they are calling these days machine learning operations that this could be a workflow of that This talk is not necessarily about mlops, but it borrows some concepts from there So first you are aware that the model that you are creating Has some some life cycle in which you have to go through all these steps that these are the ones that
We are we are expected to be knowledgeable in how can I ingest the data? How can I version it or label it and run the validation test training model fine tune it hyper parameters analysis Reading all these learning curves and then deploy and then get feedback
So this is the only the model but one of the things that I learned that is very important When you want to scale these solutions and when you are not only developing for yourself This even helps if it's for yourself because When you move to the process view of what you are attempting to do the first person that will be grateful
Three weeks later or a month later. It's yourself. You say oh, that's so good that I put it in this in this view Because now you understand All the steps and more or less how you can split the responsibilities And I put here that one of them is a dialogue contributor because one of the things I have found while working developing
chatbots with rasa library or others is that You might be the let's say expert. I don't like that word but you might be the most knowledgeable person in how to build the the the bot But you are not necessarily the subject matter expert of the topic that the bot is supposed to provide a service
And in many cases the subject matter expert is a person that they are not developers or they are no they don't have any Computer science or engineering background But they are the ones that they know what should be the training data that you fit to the bot They know what type of questions the the end users will will ask so in this way you also
Can start enabling other type of profiles in how can I incorporate their their contributions to my process to create a better bot So this is the process view That it contains many of the steps from the model lifecycle view
And what are the benefits of this when you when you do this first is a favor that you do to yourself But also you have a common understanding of this process for everyone So if next time there is an iteration that something should change now It's very easy for let's say some some new employee or or for you one month later to retake and say, okay
This is this is what I was doing here is where I am and and when you Use this process modeling as well. You can identify bottlenecks or where the things that you are doing might be failing And also this enables you with some standard that will enable iteration So you can improve with the pass of time the performance of the chat
And now like how does this look so the components view? and here This is the high level diagram of what I have architected that it works to a very decent level in which this this could be the Pilot or prototype but very decent not not poc hackathon level very decent pilot that we will go
Step by step and I will show you starting from from this part. So What is in the in this part in the left the dialogues or python code? This is your development environment. This is where you have your pie charm or visual studio sublime text Whatever here is where you have
the your code Someone can be getting dialogues In the certain format for your bot. So the only thing you do once you invest time learning principles of CICD is that you work in your machine Once you are happy with your changes or with whatever you did in your local environment you do git commit and git push to remote
So what I am doing in my workflow is that I create I make a change or in this case for this example I am adding more dialogues to the bot so the bot knows Knows more combinations and is able to have better conversations
This push will automatically and from here everything is automatic Back in the day, let's say four years ago when I started before using this You have to use the command Command line and know each one of the commands that for example the rasa framework provides to do rasa train
rasa cross validate or rasa Test data all of these things. So here what it's what is happening is that Once I commit the code Automatically using GitLab CI it will start it will start some runners or containers or it's called runner that each one of these
Will run the steps that you usually will run manually So this is not nothing new for someone who is who has been working in traditional development for years But this is more tailored for people that maybe they come With a math background or they are breaking into machine learning and they would like to build bots
This is not necessarily knowledge that they have at least for example I didn't have it four years ago when I started these these good practices of solver delivery. So Now that I discovered that's why I said oh, this is this makes my life easier. I want to share with you So these these steps that are usually executed manually they start running in different machines
So one one runner is going to validate that the data that you push To to the repository. It's okay. The format is is there with the one that they bought requires Then the the data is complete And it will train the model and it will test the model and by test the model here
Is the concept of testing in machine learning this with you can you can either create a testing test data set That you will run against your model or you can do for example in this case on my demo I do cross validation With the data that I have available. So assuming that each one Of these steps
It's successful then the model gets created So you have an artifact now and the the artifact is your model And if you are happy with the performance because when you test the model You will get some metrics of the performance of your model and if you are happy with that you say, okay So now I want to do a merge request and and I want to put my bot into the main branch and this will deploy and this will
Deploy to production or staging of the product and here this part of the paper tray. This is this is one of the the key key key steps in machine learning operations that I am borrowing and bringing it into this into this pipeline of
serving chatbots is that Whatever you do whatever you you change you make In your dialogues or in your code this by running these pipelines. This will create a paper thread and what does it mean you will have a record of For example, this is a snapshot of today the mo the model that was created today
What type of the training data it used? What was the code at that moment? So in other words, this is version control that is getting into a record That I post in a wiki and if someday you work in a company that they will get some audit And they have to show show me why your model
Was executing these predictions so you can easily go to the history of commits and say look This model had this training data and these were the parameters and here is the code and here is the data that we use to train So this is the paper trick that this is required in in in the enterprise And also if you are just playing this paper Helps you to keep track of your experiments and have some sort of report in which you can
Compare if you if whatever changes you were making was improving or deteriorating the performance of your model So once the model is ready and i'm happy with everything and all the steps and the testing From the software engineering point of view and machine learning is successful and i'm happy with everyone everything
What I do here is that using this Framework It provides a api rest That you can deploy a model server The model server in whatever you want in my case. I'm using it in world cloud and then you can use this api
api rest api to put And make it active and once the model is is is there Then it's it could be available To be tested by real users or or other people that they can serve as testers of your bot of your dialects
And but it doesn't finish there because until this point I could be quite happy that the only thing I had to do was Let's update the knowledge base of my bot Let's just do git add git commit git push it goes automatically through all of these steps. I don't have to worry about
Putting manually Executing commands and checking performance metrics because I can build a logic that if it's above certain threshold. It's okay But once you are there and the model is served In this model server and it's ready to take conversations It's very important to close the loop with some feedback and the way that i've been using this and I will show later is that
I am using the same Rest api that provides the rasa framework to extract a record of dialogues or conversations that the bot has been having with real users So what what you can do here is that when you expose it to the outer world and the bot is interacting with real
people using different Ways to say hi or to express an intent you can bring this back to to to To your environment and some sort of data playground and here you can do what you like the most that is okay. Let's use
pandas python libraries to analyze To to do nlp in other words And from there you can extract insights to later feed it back into the bot And retrain it and start continuing this what I say before This Loop that enables interage iteration to improve the bot performance
so more or less what i'm telling you is that You you could be a one person Band, but you can also bring a conductor to your orchestra someone or something in this case technology That will help you orchestrate all of these manual steps and at the end of the day by using
These principles of continuous integration and delivery will step up your game creating bots i'm very important Really? It will make your life easier. I am a testimony of that. I used to to to before as I was saying with the the other
with the other Talks I I had sometimes when I wanted to make some change and the change was minimal like adding just one more sentence Or changing it happened to us Building a bot for this topic that Have it here Have us here online. That is When we were building a board like two months ago for frequent ask questions about
Coronavirus One day we were the bot was supposed to answer No, don't use mask because mask should be used only by health professionals one day later. No, no The bot should reply. Yes. Yes, you use create make your own mask and please use it
so it was only one single change in one sentence of the bot and it will imply that you have to maybe jump into the docket image make the change many things here. The only thing you have to do is Just put it in your in your pie chart in your local environment Update it git commit git push and this will take care of that
so Uh, this is how in this case i'm in powerpoint keynote, so everything works in keynote, so I want to show you a video of how it could look like so This I will share with you also a link. This is paulic on youtube
So here typical developer workflow i'm just creating a branch in which I want to work on my changes So once I create the branch i'm using the this is the rasa format To or to to add intents or knowledge to the bot i'm just adding two words
Okay, I am i'm just adding two words here Um git add git commit and i'm here after I I commit the changes and I will push I will push them here is where the where the magic starts because I'm pushing to to remote and in the remote which is git lab is where I have configured all this
orchestration with my new newly hired conductor for my orchestra So typical typical let's say developer workflow This is creating a merge request that when I come here And I this started automatically once I did the push
Automatically this started to run this pipeline One of the steps are ensuring that when the bot is ready is because it's really ready to be deployed to production And each one of these steps what they are doing is what I described in the diagram before Is testing that the format or the template that we need to feed the training data to the bot is is correct
It's testing It's creating the model And once that is creating a model is testing the model. So Typical developer workflow with the good practices. So here what i'm showing is that i'm going one runner by runner And these are the the different machines that are executing
Automatically the steps that I that I if I don't have this I would have to do manually in command line So by using these runners I am orchestrating automatically all these stages following good practices of development and here this is how my my my workflow or pipeline
Looks like this is what the the bot The what I did what I provoked with this git commit is that is as I mentioned before these are the stages That it for this branch. They will end up writing a report that the report is to say is this what will show me that
The different artifacts that were created like these ones Different artifacts that get created when I train the model such as I don't know histogram showing the prediction confidence distribution Or a json file that is showing me also some some sort of report of how the bot
Performed based on that but this json is pretty helpful, but it's not Nice, let's say for a business person that they just want to show me show me how it looks like You know a nicer way. So when I this is step that is called write reports What it's doing is in same git lab is writing in the wiki uh The report of the experiment where you can auto generate this thing that you can later revisit or or
Keeping some track of the changes of your experiments of the or of the model that is in production So once i'm happy with everything I can merge the the changes to my bot and here When I start merging this is a view of the rasa rasa
Model server in which as you can see that moment when I when I took this video The model that was active was a model of last sunday, but here this was wednesday I'm running a pipeline that is merging the changes into the main Main branch that once this this pipeline will will finish
What it will show show us here. It's about to to finish in the deploy stage There right there what is going on? Is that? my my pipeline is Talking to the model server to rasa server and it's saying I am ready the model went through all of the steps, please deploy
And once this is successful as we can see there I will refresh this page and you can see that now the model is ready was automatically deployed and made active And from there I can use tools like the ones that rasa offer to create a link to share this newly created bot
With external people so they can test it and I can also leverage an approach of conversation driven development because from here Before deploying before following these steps that I just followed and deploying the production You can have more feedback of how the bot is performing So as as if you appreciate the the beauty of this you were working in python you made two or three changes
You git commit git push and all of these things Started automatically and to finish the to close the loop I want to show you the last part Is that in this case? I am I am using the api
from rasa and i'm bringing Okay, we have five minutes and i'm using the the api from from rasa to bring some of the Dialogues of conversations that they both had so this bot was newly created only for this event. So it doesn't have too much too many Too much data, but this can give you an idea that here the sky is the limit you can say
okay, I want to filter by number of intents, but the Confident highest confidence of the best intent or what is the type of token that people are using the most are they saying? Hello the way I thought or are they saying hello. I want this are they using To move the intent in one sentence and for example in this one
I was just extracting the last word and this for example could give me a glimpse that look If the people are saying i'm a bit mad or i'm quite furious. It's probably the pot is not it's not Working in the way that I expect so to finish that I show you how it looks like And if you adopt these continuous integration and delivery practices
It will allow you to have an orchestra for yourself with some conductor that will help you to make your life easier If you scale this up to a company this will increase communication and collaboration You can have a more reliable solution and at the end of the day It's setting you up to grow more from there
You can grow from the manual development to the pipelines that I just showed you to another point where you can Think of fully automated processes like instead of using this what I was showing you an api to call the rasa api To bring in conversations. I could automate this job every day 8 pm and run some preliminary scripts
And yeah, so this is like the the next step if you want to know more I have some links here And I will make available this started project that you can fork add from there. You can see how I I Orchestrated all of this use it using gitlab and rasa from this project You can just copy change to your pointy to your own server and it will be ready to
To replicate what I just did or I just show you in this demo with you so Thank you, I think this is I have three minutes of questions Thank you very much. William. That was a very nice talk very interesting indeed. So we have one question The
Question is that what are the cons and pros of gitlab cicd versus having a jenkins server? Okay well If you already know jenkins, so what are the pros and cons cons the learning curve? If like me you came to this, uh Cicd without knowing jenkins so gitlab is very easy to learn
So that's one thing learning curve and one one one pro is that in gitlab you have everything on there The there are in a single app. You don't have to switch environment. For example here In this I have in one single view. I have the pipelines here. I can define Find everything and what I define in my project that will be my pipeline is automatically triggered and read by gitlab here
for example, so pros and cons pro if you don't if you if you are comfortable with learning this tool because it's very easy and everything is Pre-integrated and as I say will make your life easier so you don't have to know to think about
Okay, so I need to learn a new tool. I need to learn jenkins plus rasa plus my own development here What what the reason I like this a lot is because here is everything Everything is here under this single application and whatever I push to my to my to this repository it will trigger a pipeline in my
in my gitlab ci so that's what that's something I can tell you from a point of view of A gitlab fan but also personal point of view if you already know jenkins Learning core. Okay, go for the things you already know because at the other day You just want to make your life easier and deliver faster
Okay, thank you for that answer There's a second question. I don't know whether we have time but Is there a difference between gitlab cicd and github actions? Well, yes There are different products Yes, there are there are differences that Is we can take it offline
Because I'm this is moving me into the conversation of competition Right, okay, I can I can I can I can in the in the room Sorry in the room because all of this is is public for for for gitlab We have a competition page that if you google A gitlab versus github actions in google. This is public and you will find it there
It's not like it's not possible for me to talk No, you can do it and it's so transparent that you can have it only if you search for itself, but it's a quite long topic So, okay, so let's take that maybe offline into the Into the talk chat. We posted the link on the on the talk chat in discord And and so you can then find the talk room for this. So
Thank you very much arias for the very nice talk. Let me give you your applause Thank you. Thank you