Business Dashboards
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44887 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Product (business)ImplementationContent (media)ComputerBitProduct (business)CodePoint cloudMultiplication signContext awarenessIntrusion detection systemBuildingFocus (optics)Graph (mathematics)MereologyPay televisionClient (computing)Link (knot theory)Vulnerability (computing)XML
02:18
Gateway (telecommunications)Product (business)Medical imagingGraph (mathematics)InternetworkingMultiplication signService (economics)InfinityData miningParameter (computer programming)Gateway (telecommunications)MeasurementWebsiteWhiteboardBitRevision controlSoftware testingComputer animation
03:50
Service (economics)Medical imagingComputer programServer (computing)Computer animation
04:25
Service (economics)Programmable read-only memoryService (economics)Graph (mathematics)Web crawlerMessage passingMetric systemFront and back endsQueue (abstract data type)Multiplication signServer (computing)Key (cryptography)Semiconductor memoryEvent horizonBefehlsprozessorPrimitive (album)Data storage deviceSoftwarePublic domainMatrix (mathematics)TimestampWebsite2 (number)QuicksortQuery languageMedical imagingDecision theoryDatabaseStandard deviationMagnetic stripe cardData compressionRepresentational state transferPoint (geometry)LastteilungEmailGraph (mathematics)Right angleProgram flowchart
08:23
Metric systemSound effectPlanningLevel (video gaming)Software frameworkService (economics)SoftwareMathematicsBit rateElectronic visual displayLevel (video gaming)Service (economics)Phase transitionMathematical optimizationFocus (optics)Mobile appCopyright infringementSoftware frameworkExistenceFree productPerfect groupQuality of serviceMatrix (mathematics)Metric systemRevision controlAnalytic setMultiplication signEmailMeasurementDifferent (Kate Ryan album)System callPointer (computer programming)WeightState of matterCartesian coordinate systemWebsiteSound effectFrequencySoftwareFitness functionAnnihilator (ring theory)Slide ruleCausalityMultilaterationMessage passingMappingWeb crawlerBit rateComputer animation
14:33
Singuläres IntegralStructural loadVolumenvisualisierungExecution unitIndependence (probability theory)Thread (computing)Graph (mathematics)Standard deviationLink (knot theory)Service (economics)WebsiteVariety (linguistics)Metric systemRaw image formatObject (grammar)File formatDatabaseQuery languageDatabase normalizationContext awarenessData bufferInclusion mapNormal (geometry)Graph (mathematics)ClefDuality (mathematics)Google AnalyticsGeometryComputer configurationAttribute grammarPointer (computer programming)Event horizonMereologyDatabaseLibrary (computing)Software frameworkPoint (geometry)Validity (statistics)Data dictionaryProjective planeIterationWeb crawlerRange (statistics)Multiplication signPhysical systemClient (computing)Table (information)Independence (probability theory)ResultantInternet service providerMatrix (mathematics)Metric systemTimestampOvalQuery languageMessage passingCountingTupleData analysisLevel (video gaming)Network topologyTransformation (genetics)Expected valueObject (grammar)ImplementationState of matterSlide ruleCodeThread (computing)Graph (mathematics)Endliche ModelltheorieWordVideo game consoleField (computer science)Service (economics)File formatProcess (computing)Lambda calculusSet (mathematics)Normal (geometry)Connected spaceDimensional analysisSequelStreaming mediaFunctional (mathematics)Element (mathematics)Instance (computer science)Uniform resource locatorChainReduction of orderSelectivity (electronic)TwitterMathematical analysisInsertion lossTraffic reportingAverageRow (database)MereologySocial classDifferent (Kate Ryan album)Fitness functionEmailProof theoryPrimitive (album)Analytic setInternetworkingWritingStandard deviationReal numberStructural loadComputer animation
23:36
Service (economics)Bit rateGraph (mathematics)Graph (mathematics)Instance (computer science)Metric systemNumberEvent horizonBit rateWeb crawlerComputerDecision theoryInjektivitätMultiplication signOpen sourceService (economics)Analytic setSoftwareBasis <Mathematik>Connected spaceSuite (music)Query languageMaxima and minimaCountingElectronic visual displayInterface (computing)2 (number)Food energyText editorUser interfaceDatabaseQuality of serviceConfiguration spaceFront and back endsData conversionPhysical systemReduction of order3 (number)SequelWritingObject (grammar)EmailWeb browserAverageDebuggerStructural loadDivision (mathematics)Right angleGroup actionArithmetic mean
27:40
Task (computing)IterationNP-hardPredictabilityComputerServer (computing)IterationProduct (business)Process (computing)Task (computing)Computer-assisted translationScheduling (computing)Phase transitionComputer animation
28:40
Computing platformBroadcast programmingDistribution (mathematics)Operator (mathematics)Module (mathematics)Hash functionOpen sourceExecution unitWebsiteEvent horizonPasswordReading (process)Computer-assisted translationWebsiteDifferent (Kate Ryan album)Set (mathematics)Point (geometry)Row (database)Medical imagingDataflowLevel (video gaming)Task (computing)WorkloadGoodness of fitBitDatabaseAuthorizationService (economics)ComputerProcess (computing)CASE <Informatik>Multiplication signProjective planeConfiguration spaceScheduling (computing)Direction (geometry)Product (business)Integrated development environmentObject (grammar)Virtual realityMetadataConnected spaceData loggerPhysical systemOpen sourceDirectory serviceDivisorCodeTraffic reportingEvent horizonUser interfaceCartesian coordinate systemMeta elementOperator (mathematics)ResultantFunctional (mathematics)Computing platformComputer architectureReal numberComputer program1 (number)CubeGraph (mathematics)Structural loadComputer animation
34:46
Constraint (mathematics)MeasurementCodePlanningMultiplication signSoftware developerResultantProcess (computing)CuboidBitHypothesisBeat (acoustics)Sheaf (mathematics)Complex (psychology)Computer animation
36:54
Factory (trading post)Line (geometry)WebsiteScaling (geometry)FeedbackFactory (trading post)Goodness of fitProduct (business)Content (media)Digital rights managementCodeReading (process)Different (Kate Ryan album)LinearizationLink (knot theory)TwitterFeedbackOnline helpWeb 2.0WebsitePhysical systemRevision controlExpert systemInformationAnalogyPresentation of a groupProcess (computing)Numbering schemeScaling (geometry)DataflowComputer animation
39:21
Computer architectureVulnerability (computing)CodeNumberBitDiagramMedical imagingType theoryLink (knot theory)TwitterComputerMultiplication signProduct (business)Information securityFeedbackTelecommunicationDistribution (mathematics)Graph (mathematics)Mobile appPhysical systemConfiguration spaceDigital rights managementLocal ringScheduling (computing)Moment (mathematics)Message passingBit rateResultantLoginDataflowCellular automatonVirtual reality1 (number)Musical ensembleArchaeological field surveyComputer animation
43:09
Constraint (mathematics)Software frameworkQuery languageWebsiteEvent horizonPasswordOpen sourceModule (mathematics)Operator (mathematics)File formatHash functionRevision controlService (economics)Medical imagingDataflowDigital rights managementProcess (computing)BitData streamComplete metric spaceEndliche ModelltheorieGraph (mathematics)Shared memoryVirtual machineScheduling (computing)Square numberComputer animation
Transcript: English(auto-generated)
00:07
Thank you very much for the introduction So yes today I want to Tell a bit of a story of how we built business dashboards for for a product I am indeed the creator of vulnerable, but I want to talk just about vulnerable
00:24
I will show different tools and so yes using vulnerable Afro and Grafana all together to build really quickly some business dashboard Before we start so I'm Roman Delgue I'm Making computer related stuff since quite quite a long time now and this year my focus is I decided to change
00:46
Quite a bit what I'm doing, and I'm building a new company called maker squad My goal is to build cloud native products mostly sass products using containers and stuff like this
01:00
for ism myself and live on subscription and or is our clients and And yet just help them achieve this This cloud native stuff, which is not really natural for a lot So there is a lot of stuff today I'm sorry if I'm go too fast sometime But I will give you a sidekick link after with all the all the code and everything
01:26
The idea is I will introduce the product so you you know what we're talking about It's a bit of background context, then we'll see how to plan what we want to dashboard eyes And we'll go into the technical part which is implementing the data pipelines using vulnerable visualizing it using Grafana and
01:45
running seriously in production with airflow and the bit of Well what to go from there after that, but yeah that will be it So it has been said, but I created Bonobos of so I'm a bit biased here
02:01
I'm sorry about that I'll try to be an objective But make your own opinion if you're building a product you're bringing anything make your own research take a lot a lot of time in Planning it's very important, and yeah, I recommend nothing, but maybe it's an idea You could try to apply to your your own products and business
02:20
So the product It's a pretty simple thing It's something that you give a neural a size and it returns an image It was in fact of a service that a friend of mine Developed a long time ago like 10 years ago an infinite amount of time in the internet age
02:41
And it was living like this in January If we time Delta years 9 I added a parameter to time Delta here We Come to February where I decided that it was not making money it was Using a lot of time wasting a lot of time and it was time to shut down the service and at this time
03:01
I called him and we agreed on me taking over the service with brand new thing So in March he started to send me his traffic, and I didn't think he had this many traffic So I started serving gateway timeouts, but after a bit of work a lot of work actually I could have the new version of the of the website with which was no PHP anymore or Python
03:26
And you'll see after our talk a few months after I started to measure everything technical It's the background to be able to measure business things after and in July Also known as last week literally we launched the private beta
03:43
And started to win on board all the old users there was Yeah so things Did not go exactly as expected, but I'm pretty happy it was a nice problem to have too much traffic Actually when I measure the thing I could say it was around 1 million
04:03
Images served by day, which is not a lot if you're Google, but which is a lot if you just have few service somewhere And yeah, so nice program to have so
04:24
Just so you understand how it works that's Basically the world service at least the world user facing service It's pretty standard on the website It's there is a load balancer that sends traffic both to a Django website
04:40
Which is the marketing website and the user accounts? Manage the API keys the billing or stuff like this and Sounds also the traffic to the API server if it's an API query which is written using tornado Not really mother and not really sexy, but it's really fast, and it really works well for us This API server basically just have to serve images as fast as it as it can
05:05
Whenever an image is not here which happens quite often It sends a miss event to a message queue a rabbit MQ message queue Another service called janitor picks it up decide update the status first decide whether or not this request is legit as this
05:22
Domain not flooded is this is this some kind of abuse etc And if it decides it legit it sends a crawl message to another message queue a spider picks it up open the image Save make a screenshot resize it compress it blah blah uploads it
05:41
And at the end it sends either created or failed message to the Evans message queue Which will be which will be picked up by the janitor once again and the janitor will update in ready So the API server can serve the request next time and user asks for it basically, that's All the service is what I said
06:02
But to manage that we also have a lot of data producing services in the back end I won't detail all of those services but mostly we have prometh use which is a timestamp oriented database that's course metrics from services and stores them into a bundled timestamp database of TSDB and
06:23
also provide the query language that amongst other graph and I can Ask can query and get metrics from so build graphs from that And as it's a REST API like HTTP API you can also Just query it yourself and really use it to get some data
06:42
Very important to we have external services Which is pretty much services. You know Google Analytics right Milgan etc And that's interesting because there is a lot of data produced here, too And mostly the most interesting insight we can get even if it's not Not any anyone in big data if it's small data
07:02
But the biggest insight we can get is when we cross the data from internal things with external things So at this point I already have Dashboards of technical metrics and I insist on this technical world because it's not what we want to do after But we already have those dashboards made in Grafana
07:23
It's really easy to build you just write a query and you have the graph. It's very quick to do We have this CPU memory network like standard monitoring thing and we also have also standard monitoring thing But more Related to one service like nginx is returning
07:43
The well the amount of 200 300 400 500 we have per second written by nginx the timing of request So we can know whether or not things are going nicely I Insist once again on the fact it's technical metrics because there's a lot of data every 15 seconds per material scores
08:01
Everything and and and stores new data But we don't want to store forever this data because we can't make decisions based on that. It's just that that helps us restore the service whenever an incident Arise or just react to what's happening. It's it's good to know what's currently happening in the service
08:21
But it's good to know we have that before we do more Also planning for many. What are you drinking a conference is very important so the plan is Basically based on that if you can't measure something. It's really hard to improve it
08:45
You're just blind blindly running in a thing as it's very negative I like better the the other version which says what gets measures get improved It's not as easy as that because there is metrics that you can look at after as much as you want and it's just
09:04
Consequence and effect so You can try to improve it, but if you don't fix the cause it wants, but if you choose your metrics wisely choosing cause metrics Really having a focus on yeah, that's that's our goal. That's our team goal. That's what my goal Will really help you improve that and of course there's not one answer to this question
09:24
The answer which is not an answer is that you should not focus on vanity metrics for example sooner I said there is 1 million requests on the API per day. That's a perfect example of example of vanity metric It's good to say yeah, we have some traffic. It's good to say to your user. We're not
09:42
We're not a tiny service But if you focus on that it will just waste your time because I could serve 1 billion requests. It would be just costs If none of those requests are actually built it it doesn't mean a thing So to plan the metrics we want to
10:02
To measure It's important to consider the business you are in like we are in software as a service But there is different metrics for different kind of business. I will give you some pointers to find The state the state of the art and in this state in in this regard after that
10:22
It's also important to not look at the same thing if you're like early stage business pre revenue pre market fit Pre-product market fit or if you're in a late-stage business you won't look at the same kind of thing Hopefully a lot of people are really smart people built frameworks for that And that's I'm speaking about business frameworks and not technical frameworks, but one you may have heard about is the well-known
10:47
Framework also called I don't know if I say that correctly Also called the pirate matrix framework, which says okay? Users are complicated and we will segment the user the user journey
11:02
within our service into five kind of steps maps to when we attract new users when we get them to Give give us a little something like when we activate them for example It could be they leave the email that could be they create an account them
11:20
But that could also be they give you a phone call or maybe they download your application if you're in a mobile app Then you have the retention phase where you say okay? That's good You do I know who you are, but now I need to have you really Come and come again need me every day or every period of time you you defined I
11:41
Reversed the two and thing I didn't do it on the slide, but the then you have the revenue how do you get money from the from this user and That's not necessarily. He pays directly it can he can generate money and directly also and you have the referral phase that I put really at the end because
12:02
In fact for user to recommend your service. He must be like deeply in love with your service and He must love your service so much that is willing to actually risk something of his social existence to Recommend to his friend so for example to explain that
12:21
People recommend slack because they think slack makes them like look cool But if you a pass site or some strange service somewhere People don't think for now that you're cool, so you can try to make referrals, but that won't happen at this stage Another way to present the same thing is the ash Moya version
12:43
You will you will have all that after And If you want if this is not enough There is people like Alistair crawl and Benjamin you scovitz that works in their book in analytics on Much more detailed version of the same thing actually it's the same thing you can look for the different
13:06
A a or phase on this, but they are all here But then it's much more finely grained and you can find Mmm. It's fine-tuning. It's it's optimization. It's so as it's optimization. It's for later
13:22
For today the plan is to focus on the activation phase like let's say acquisition to activation phase And we'll just use oh, that's enough So to summarize what business software service its stage like the defining in Linux, which is
13:42
Very early stage we need to know if the if the People are because we have users, but we need to convert them to members So do we have enough do we trigger enough emotion to make them want to become members? Do we trigger enough?
14:02
Need answers that Actually, they stick to the service and come back and use it and use the AP cage So for now because I don't have a lot of time we will focus on rate from acquisition to activation and Quality of service both because we want to measure it to improve it and both because it's the main
14:25
Testimonial we can have if we are transparent and our quality of service to users They may think we are actually serious on how we doing it so first of the
14:43
First of the three technical steps is actually using bonobo to implement some data pipelines to integrate data in some metric database Very simple we get some data We aggregate it eventually normalize it and we put that in a metric metric value database
15:03
Most probably timestamps. So I thought about the most simple model I could find about that Actually, it looks like this. It's metrics hourly values daily values That's very quick to write
15:21
probably not the best but it works for this stage and it's important to note that The technical metrics yield a lot of data. Those metrics will yield not a lot if we have one Data point per metric per hour. It's 24 data points per day. It's really not a lot so
15:41
That will be enough if you need to build a much bigger system, of course There is star and snowflake schemas. You can find a lot of literature on the Internet And yeah, so Just a quick introduction because I guess you may not know bonobo as it's really young project to you Less than two years old now, but bonobo is an extract transform load framework. The goal of this is you build
16:08
Assembly lines of data transformations We've independent stages and you will pass rows of data from one stage to another so less metaphorically you will use
16:20
Okay, it's fitting You will use different Codables and iterators and and classes instances to get some data and here we are Selecting some data from the database. We're qualifying it. We're joining to other database and sending emails The point is this the send email function will run while the select function is still yielding results
16:44
So the first report will be sent and maybe select as still 1 million were to select It runs in independent thread that I spas first in first out it's kind of kind of stupid It's of course
17:01
I have my example was linear but it supports any kind of directed acyclic graphs and we'll see that after Imports and it's standard Python so you can use it in bonobo or outside bonobo the same code. That's pretty neat and Getting started is three lines and if I had a long sentence you could have done it before I finish
17:24
So let's write our jobs once again apologies if the code is if I'm going too fast for the code It's why I put up on URL Upload the code and notify as soon as it's done at the end, but first extractor looks like this
17:41
It's an object count reader. So this object count reader Builds a sequel query which is rather simple it counts objects in a table and formats that as a tuple of dictionary of dimensions and dictionary of metrics Of course the query is parametrized
18:03
so the person zero Needs to be filled and for that we'll simply use a dictionary. That is a dictionary of Table to metric name and we'll create An iterator using the items the dict items built in
18:22
That will pass data first in first out to object count reader that will then Send the request to the database and etc. We'll see more when we aggregate all the thing The normalize is not a real normalization or there is no validation here But it will do the job for the proof of concepts. We will just say okay
18:43
Whatever comes here is I need two fields and first one is called dims to control is called metrics We don't validate anything, but it's really easy then to replace this simple stupid thing with Real schema validation of whatever is inside if you need to be more solid
19:02
We need something that writes to the metric database so I wrote one that Can write both to the hourly values and and and daily values tables it filters out the rows it doesn't need and then
19:20
it's not really important to get the details of that, but Simply it will be called on each each dimension metric tuple, and it will put it into into the database And of course we need to compose all that which is probably the most important part so we instantiate or normalize thing at the top We create a graph instance that get the readers
19:42
We had before so the dict items on the object count reader Then we normalize it just instance above and we had two chains We have two instance of analytics writer, which will filter out non hourly metrics And insert the hourly metrics in the database and the other one will filter filter out our new metrics and insert
20:04
daily metrics into the database Because it's better when we visualize it. I skip a slide that will do that so It will be passed first in person between all and after the set fields The data will be passed both to a daily analytics writer and both and to all your analytics writer
20:25
And it's their role to filter out the rows it doesn't want Just before we passed the database connection implementation, so it's SQL alchemy databases in engine park, sorry
20:42
But yeah, not of course you need to connect You can run it you will have linear status, but it's not clear. Okay. We got it Let's add a lot of readers again, I'm running quick, but You you'll get all that we connect to Google Analytics, so we provide a service we call Google Analytics
21:03
So of course we will provide a client implementation In the services dictionary, but here same ID We send a query to an API and we yield a tuple of two dictionaries dimensions metrics We can call Prometheus you do not have to know what's the API of Prometheus
21:23
but you have a query range and points we query this thing we ask for some metrics and the one we need we Often it's an aggregate. It's a an aggregate query. So we just take howly averages or things like this and Again, we yield two dictionaries dimensions metrics
21:41
We have spiders count. So we have a lot of spiders running some are inactive some are active So we get all this thing and we'll use a concept similar to funk tools that reduce in Python but adapted to the stream processing thing and we'll use so we'll use a reducer that will
22:01
Take elements two by two and just make one So we take the spider statuses and we make a dictionary of counts of active and inactive spiders Which looks like this so the you need to initialize to give an initializer to the to the register thing and
22:20
The function to reduce so everything that goes out of spiders with the status of spiders Goes into this reducer two by two and as soon as it's available It's yielded and we use a lambda here to just format to the dictionary of dimension dictionary of metrics. I
22:41
Fear here, but the results graphically looks like this if we Put all the readers At the same time in the same graph and this graph can be executed you don't have to execute everything at the same time, but this graph can be executed and we have some kind of bigger statues
23:01
If someone knows how to write ASCII art trees in console I really love to discuss the algorithm because for now I can't I Again I'm Bayezid, but it's really easy to replace parts and that's That's that's really what what I like most is that for example the normalizer is still this set field thing
23:22
Tomorrow I will use maybe something like suburbs or any schema validation library to to be sure that's my My data is in the right format enough talking about Bonobo We Need to visualize things and we already have a graph and I instance installed
23:44
So it will be quite easy to just take some of those metrics and make graphs with that So I guess who doesn't know graph and at all okay, so Grafana is quite a old software. I think it was a fork of
24:02
I know the suit like this before and the goal is to connect data sources and to Graphically configure graphs by making queries. So the interface to configure a graph looks like this You just have a query editor. You put the query and you get the graph. That's pretty much as simple as that
24:22
of course if you don't have data, you can't visualize anything but or all the configuration is done in the web interface and so for example, this graph is one of those we use for the curious thing Which shows how many events we got every hour about how many miss we get we got how many created event
24:45
We got how many failed event we got so we can compute ratio between created failed for example Etc. We have also the one that passed through the reducer thing is the number of spiders
25:00
So when we launch more we see the number of inactive spiders active spiders We can walk we could walk on better display here because it's what means inactive. Maybe it's just resting so anyway But much more readable for users. We have this Graph of how many times does it take to get a new picture?
25:24
At the time I took this graph on average. It was 16 second If probably you don't have all this advertising things JavaScript thing that make the browser load very long Maybe you were in the 9 seconds range, which is the minimum we get here
25:43
So we use that also the same kind of source to be low public dashboards here The front end is something based on c3 And that's just a parenthesis not very important, but that's how we show to users the actual status But interesting also is that as you can write sequel queries directly in Grafana you can
26:08
Sorry You you can actually make computation directly using your database engines engine so here. We had user counts We had new sessions from analytics so user counts coming from the count of objects
26:21
That we computed every hour before new sessions that's coming from Google Analytics API calls and you can just divide one by the other like subtract the maximum on a day with the minimum on a day of number of users then make a simple division and gets the conversion rate between
26:42
acquisition and activation Here it's a bit strange because the we have a that's actually real data, and we have like a 25% Conversion rate at the beginning So you have to cross us also with Evans because it's the day when you we Mailed every people that said hey, I want an account
27:02
We sent the the mails this day, so of course the this kind of acquisition right is not possible normally Unless all your traffic actually already said they wanted an account That's very early as you seen in the
27:21
In the timeline and action at the beginning That's the affair of the last month and last weeks of of work, but here That's that's our basis for the for Seeing a lot more thing and we really want to base our decisions and data So that's the basis to make the service evolve so
27:50
Until then that's pretty good. It runs on my computer. It runs on the production servers, but This iteration zero, let's say
28:02
Is yeah, it's it needed to work real quick so a cron job was running everything every 30 minutes when something fails actually we could know if something fails because it was Kubernetes cron jobs, but It's you need to you need to have a look maybe if you have a lot of job that fails
28:21
You don't know which one failed Some expensive tasks were running every 30 minutes It was hard to run manually so because you have to export the cron job into a job We schedule that into the into the scheduler, and then eventually delete the job afterwards so a lot of things And we wanted to do To the better on this side so my cats
28:44
Gently proposed to handle that I can declined the proposal and We Installed a flow To manage that so as you may know a flow is a platform to programmatically author schedule and monitor Workflows that the official docs, and I think it's a pretty
29:04
precise And we mostly use it to schedule and monitor here Once again who doesn't know a flow at all Okay So it's a project that was created a few years ago by alien B
29:21
It's now on a project under the Apache incubation program. So that's a really great news for the future of this and The role is to schedule and monitor jobs Either we've like if I run it on my computer to just get your jobs on my computer directly Or if you're running on a cluster for example Kubernetes
29:41
But also kind of cluster can walk to you can distribute workloads using celery desk or Even they are working on a Kubernetes executor, so soon it could even submit Kubernetes workloads directly And it can run pretty much anything a computer can run Python being a special case of anything a computer can run, but it can run anything and it's written in Python
30:06
So the configuration is actually Python code Without Too much details the architecture of airflow, which is important to Well, if you want to run a flow you need to understand a bit the architecture because otherwise it will be held to to run
30:21
But there is a web interface and which was just after that helps you Updating the metadata database and there is a scheduler service that reads the meta meta data database The data is hard to say With that and just say oh, but say I have a task to schedule so let me find a worker that us
30:42
Okay, you can run it so it's on the workload to worker get the results get the log file and update the metadata with that For us it looks like this so we all the tasks we configured before we defined it as dags or Directed acyclic graphs, which is the name of the task in in a flow
31:06
And My timer shut down, okay Which is so a task in a flow and here we could separate For example the cleanup task at the very top is only run daily because it deletes a bunch of rows in database
31:25
No need to run it every hour and all the rest is run hourly Which is what is great is that you can get the log files of one individual task you can get You Can get all the all the runs you can run one manually you can know the timings of different individual tasks
31:43
So as we plan to add much more. It's great to have and we also plan to have Dependencies between tasks because the main goal of a flow is also to say okay I need to run the five tasks here and once all they are all are done Let's run this thing and then dance then send a report
32:00
Bonobo would do on one data sets first in first out at the row level a flow does that at the job level and That's something you find in. I mean legacy ETLs like talent Pentaho things like this you you often can manage the workflow between jobs and between tasks here. We use two different tools
32:26
Real quick the configuration so you build DAG objects here we build Simple DAG object with just one operator which runs Python code in another virtual environment
32:42
We built a lot of DAGs using this function dynamically so for each metric each data source We build one DAG and we build at the end the clean all which just yeah delete things from the database One thing airflow managed to is connections, so here we created a person events and a person
33:06
website connections in a flow in the directly in the web interface and we use Probably suboptimal trick to just use those thing pass it using the system environment And so our application that already
33:22
Respects pretty much the 12 factor principle can just use the database Connection from the environment and we can just configure it in a flow There was a bit of question we had to solve we needed to know where to start the
33:40
DAG so we can run locally the same as production we decided to To put it in directory in our only code base We have one code base that runs all the different services with different entry points, and that's one more three points So we built an image Not important
34:02
It was probably a bit Complex to set up at first probably also because we didn't have a lot of experience with that We finally found that the helm charts in the community are not really good, so we found a company called astronomer. That's
34:21
That's I think they provide Air flow as a service and they built really good quality images Helm charts, so Helm is a packaging thing for kubernetes yeah, the last thing also is that we had to read a lot of the air flow source because Yeah, probably I will try to contribute a bit to the documentation, but there is a lot of thing that lacks into documentation, so
34:46
prepare to read code Which is also true with Bonobo and not with Grafana which is really well packaged So from there I have about five minutes left from there
35:04
Plan n plus one is pretty much the same as plan n except that it's really easy to lose a lot of time When running this kind of experiment so what I I'd suggest is that Every experiment you do on business data is
35:21
Should be time boxed for example as you ten bucks for example sprints in in development process You should time box experiments on data and decide before and what are the results the numeric results that you actually? Define a success and define as failure or not success
35:42
And saying yeah, I will measure that in two weeks is important because otherwise you could say oh That's not yet or objective. Let's wait a bit. Let's wait a bit and probably maybe two years after you're still there Again Ash Morya built some simple Canvases apparently canvases are easy to sell so they but that you should use something like this
36:06
To write before and on the paper While you're trying an experiment what x what hypothesis you you need to prove wrong? Also, what kid what kind of line in the sand you you you're putting down and
36:22
And yes, so one week two weeks one month after you can just say okay. Do I did I validate something yes? No, what's the next section? Yeah Yeah, I would probably move on a bit on the text side yeah We have we want to show in Grafana month-on-month data
36:42
Very important to see the movement We have a lot of ideas We will probably use a very complex process Yeah, just so just to finish that
37:05
The To go back to the assembly line analogy Airflow is really a very good factory manager I mean you you you you're building a lot of different thing It it doesn't care about the jobs content. It can't help you building dependency between small teams
37:26
And some of you assembly lines could be built with bonobo Maybe once again make your own research and buy is it so I don't want to advise something. I can't do A lot of the work here is based on very good books, so I need to make
37:44
Advertisement for things I have no interest in Like site reliability engineering, which is the best book ever you can read about managing systems in production linearity when where you can find all the schemas for different kind of businesses like the one I
38:04
shot at the beginning and scaling lean which is Book by Ash Maria whether the schema of the different canvases are coming from but if there is one you should read It's site where they beat engineering and it's free you can you can get it from Google directly on the web Of course if you're on the paper version is paid
38:24
I Really like feedback so I said before my I was frightened to send too much information here So I really I'd really like to know what you thought about this presentation so for that I will provide a link which is the same link as
38:41
Where you you'll find all the code example as soon as I can upload them so probably this afternoon or this evening And I will announce on Twitter too and this weekend Saturday Sunday I will announce it again in the sprint announcement session, but there will be a sprint and bonobo If you want to come just feel free it's very open you don't you don't need to be expert in anything you
39:05
beginners experts Python is thus non Python is thus everybody is welcome and even if you don't want to go to bonobo ETL sprint You should really consider sprints, which is a really nice way to learn things other sprints I don't know which one dies this year, but really good thing
39:25
All the resources will be available on this link You can I will announce on Twitter as soon as it's updated with the code and there is a link for feedback so That would be really nice of you if you can just send me a line that was stupid that was good
39:40
That was I would have added this or this was hard to understand. I really like this Yeah, basically that's it so thank you very much, and if I don't know if you have time for questions, but Thank you very much for this very interesting talk
40:02
We have time for a couple of questions But I also want to remind one thing if you downloaded the app there is also a way to rate all the talks Please do that give five four three whatever stars for all the talks any questions
40:29
Hi, thank you for the talk So I Wonder about the airflow deployment you have been using some what type of a securer
40:42
So the executor executor yes Local or salary executor you can talk a bit about the deployment, okay, so We used both So let's start with my computer
41:01
On my computer. I just went with the local executor, and it's pretty easy to have it's hard to get the logs because there is a bit of communication problems, but I So exactly I build the docker image with both the airflow code base based on the astral number distribution of airflow with the opposite code
41:21
Bundled in a different virtual environment, and I just want the docker image on my computer To deploy to production the same docker image is deployed to a Kubernetes cluster But the configuration is using the salary executor so the salary executor Are you familiar with salary a bit?
41:46
Okay, so salary as a Like manager thing I don't really know the name But that runs in the scheduler of airflow and that will just handle sending messages and receiving
42:00
results from the workers so The deployment is on our side is made using the Kubernetes Recipes that we got from astronomer we changed it a bit. I can share that also if you should need
42:21
Mostly the struggle was to understand the architecture. It's it's why I did put the airflow architecture diagram here because as soon as we really took the time to Understand what was happening? Everything came in pretty logical But as it's a distributed system just trying to put a thing could
42:44
Create some WTF moments about that Does that answer your question? Okay, hello sec next question. That is maybe a little bit related How do you actually combine airflow and vulnerable because I guess both have
43:06
individual graphs defined Okay, so actually it was Done Here we actually run another Python process from the workers in
43:22
in What a flow workers will run Python process that just runs graphs Again, I'll share this so probably you you can just take the text version, but In fact, we really use a flow to manage the execution lifecycle of the of the job and we really use vulnerable to do
43:45
the actual work That's not the only way you can use a flow There is also ways you can pass that are from one job to another but the dependency Manage management of a flow is more Is more gearing towards
44:03
Waiting For things to complete because we know we need a completion before we learn something else while bonobo is trying to do As soon as available. Let's continue the pipeline. So it's more a streamline Bonobo is modeling data streaming. So both are combine able
44:26
Thanks for the talk just a curiosity question, how many machines do you run this on Exactly three today. Okay, and do you deploy with what? It's actually deployed using Kubernetes
44:42
so we have a Kubernetes cluster if you not don't know Kubernetes is a bit out of topic, but Basically you define manifest of your services You just put the manifest on Kubernetes and the scheduler in in Kubernetes runs the Docker image you define in your manifest No, no, we use a managed version of Kubernetes on
45:05
Big vendor whatever. Okay. Thank you very much again