Grafana - Past, present and future

Video in TIB AV-Portal: Grafana - Past, present and future

Formal Metadata

Title
Grafana - Past, present and future
Title of Series
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English
Production Year
2017

Content Metadata

Subject Area
Abstract
Why was Grafana born and what problem is it trying to solve? How do we try tosolve these problems? We will look into some of the recent changes in Grafanathat makes it easier to create, share and work with your dashboards. I'm alsogoing to look into the future and talk about how we see the future of Grafanaand talk a little bit about of software design philosophy.
Loading...
Graph (mathematics) Service (economics) State of matter Projective plane Time series Bit Database Cartesian coordinate system Mereology Internetworking Personal digital assistant Data structure Metric system Near-ring
Point (geometry) Server (computing) Service (economics) Sequel Execution unit Source code Time series Login Mereology Front and back ends Revision control Heegaard splitting Frequency Strategy game Different (Kate Ryan album) Single-precision floating-point format Elasticity (physics) Endliche Modelltheorie Plug-in (computing) Physical system Point cloud Default (computer science) Graph (mathematics) Theory of relativity Information Weight Database Cartesian coordinate system Web application Process (computing) Personal digital assistant Mixed reality Blog Website Point cloud Metric system Table (information)
Ocean current Common Language Infrastructure Slide rule Default (computer science) Statistics Server (computing) Group action Dot product Graph (mathematics) Information State of matter Counting Line (geometry) Automatic differentiation Number Revision control Mathematics Personal digital assistant Electronic visual display Plug-in (computing)
Serial port State of matter Information overload Multiplication sign Source code Range (statistics) Execution unit Set (mathematics) Function (mathematics) Electronic visual display Series (mathematics) Information security Physical system Area Mapping Keyboard shortcut Maxima and minima Bit Instance (computer science) Variable (mathematics) Regulärer Ausdruck <Textverarbeitung> Type theory Arithmetic mean Cycle (graph theory) Asynchronous Transfer Mode Statistics Server (computing) Service (economics) Link (knot theory) Real number Time series Code Computer Event horizon Thresholding (image processing) Product (business) Number Template (C++) Power (physics) Goodness of fit Average Profil (magazine) String (computer science) Gastropod shell Software testing Time zone Default (computer science) MIDI Dot product Graph (mathematics) Information Graph (mathematics) Planning Database Total S.A. Line (geometry) Software maintenance Timestamp Unicode Uniform resource locator Integrated development environment Personal digital assistant Query language Table (information)
Content delivery network Service (economics) Serial port Code Multiplication sign Source code Range (statistics) Time series Web browser Thresholding (image processing) Front and back ends Number Revision control Strategy game Computer configuration Average Different (Kate Ryan album) Reduction of order Website Message passing Plug-in (computing) Default (computer science) Multiplication Graph (mathematics) Debugger Planning Bit Query language Personal digital assistant Iteration Metric system
Personal digital assistant Statement (computer science) Endliche Modelltheorie Resultant Condition number
Open source Multiplication sign Source code Time series Code Product (business) Cuboid Diagram Plug-in (computing) Physical system Area Multiplication Graph (mathematics) Mapping Weight Projective plane Shared memory Web application Category of being Uniform resource locator Query language Telecommunication Data center Website Pattern language Whiteboard Metric system
Suite (music) Group action INTEGRAL State of matter Multiplication sign Source code Front and back ends Formal language Array data structure Hooking Different (Kate Ryan album) Synchronization Computer configuration Matrix (mathematics) Electronic visual display Email Mapping File format Fitness function Bit Instance (computer science) Variable (mathematics) Type theory Process (computing) Internet service provider Self-organization Right angle Freeware Server (computing) Service (economics) Computer file Time series Field (computer science) Twitter Template (C++) Number Revision control Latent heat Goodness of fit Causality Selectivity (electronic) Booting Proxy server Plug-in (computing) Authentication Multiplication Graph (mathematics) Demo (music) Planning High availability Database Personal digital assistant POKE Point cloud
Digital electronics Graph (mathematics) Code Patch (Unix) Multiplication sign Source code Moment (mathematics) Shared memory Time series Database Representational state transfer Web browser Cartesian coordinate system Proper map System call Twitter Wave Self-organization Website Musical ensemble Address space
and I'm here the talk about Capano some history some
current state and a little bit about the future for those from you who don't know grow fauna by clicking the simpler why is that okay nice Griffin is dashboarding tool for whistle icing time series data it's not a database and it's not a collectors so only which lysis data and in latest release we also alert um data I'm not sure if the lighting is okay this is gonna be fun so yeah the most common use cases for graph onna are visualizing home automation in which lysee mitri near in home automation application metrics as what its users doing not what is the service doing intent of things in Ruston sensors and also infrastructure monitoring the intention of Ravana was not in trust structure monitoring but that's the most common use case by far a small disclaimer I was not a part of the project from day one but I worked with the Tor killer the world who is the creator of the project at the same company when he made his and I joined kind of about the year after he decided to leave that company so I can't take credit for everything the reason we create a he created a graph ona was that we didn't have any good metrics
dashboard after former company the the former company's infrastructure started to look something like this we had a process of splitting a big monolith into smaller services I don't want to say micro services because this was like five years ago so it wasn't made up yet and also there wasn't super super tiny we had around 65 or 70 deplorable units on a bunch of different servers but there's nothing compared to the madness of tray anyway to grab and to get an overview of the system once we have split it up so much was very very hard and Long's just didn't carry me more we started using the LX tactful logs and we like that for logs but still it's not possible to get the complete overview what your users are doing right now so we started using graphite for user data not anything regarding servers we still still had our proprietary monitoring system doing checks on all servers I don't wanna mention the name of this service but that wasn't the main use case user metrics was the main use case so that's why we developed graph on I
inserted the way in the future from the beginning and the initial version of graph owner was a front-end only application it didn't have any backyard it was just a web application asking for years on dashboard from elasticsearch and when you save them you just send dashboard as a yes new blog to elasticsearch and that's it tandem simple start which was kind of which was good because it got a starting rather than doing everything perfect eventually we got support for influx to be an open theist me in Griffin o1x and we started having more panels we got the single start panel and of course the graph panel that was the main parts of one X into X Griffin I got its own back-end everything go of course cuz that's what you do the right stuff you go and we decided to use a relational model database to store all the dashboards instead and the only reason for that was to make it really simple for people to get started so by default - I use a sequel Lite but you can switch it to my sequel of postcodes or something like that we really wanted an easy start experience for new users Griffen also got an API at this age for automated automation and so on it got we introduced users permissions and so on and this was also about when I decided to quit my former company and start working full-time with a performance we talked in Ravana 2x we have the three new data sources from meteors elastic in cloud works and we also had a new panel called table panel and around this point we started thinking more more MORE about the plug-in system because we had a rough plug-in system in 1x for data sources but now we kind of realized that we also need a plug-in system for panels and we also know that everyone was creating time series databases every startup strategy included time series data bases at this point and we also added a colouring amazing and then support for mixed panels so you can have you can mix different data sources in one panel and we found this really useful when aggregate were one of you infrastructure data with user data and even business data cuz they tended to be stored in different databases so we had some information in elastic some information in graphite and some information in no that was it another system that we had custom-built data sources for and mixing them is really what was really important for us because it gives much better overview when you have them in the same graph rather than two panelist side side and I we catch stuff that you can't just catch stuff without it so much easier to see stuff and find a problem rather than trying to find a problem by scrolling and comparing a panel side-by-side in Griffin 3x yeah but we need on four we need them for the camera so raffinate reacts so anyway granite reacts we may we rewrote a lot of the UX to make it much much much more prettier your phone or 2x was kind of rough we didn't feel like the design and UX is important for us cause having people or dr. Guevara is one of the key strengths we think at least compared to other dashboard tools we think this is the most beautiful one and it's the easiest one to get started with we know it's controlling the edge said sometime but compared to others it's steamed much easier and inclusive we created nice so we also created our companion site food graph and I called graph on a golf net where you can host plugins and dashboards so you can upload your data sources and you can upload panels and so on and this was a major rewrite from our part because we had to stabilize the API for a long period to enable the plug-in system but but there was no big feature release for the end user per se but plugins should enable end-users to do
more stuff we also created a CPA CLI for managing the plugins plugins and that's was a kind of enough history I think it was the major changes and this is the current state of Nirvana so these numbers are public on plague refer not org I don't think you can read that URL but in the slides later and every day every group on a server sends some statistics about user count at ports and so on there's no sensitive information in that home feature they're very it's very important for us that people can feel safe about using it but it gives us so much nice data like versions and so on so you can see new features taking taking off who don't taking off or stuff like that so it's very important for us
the graph panel is like the bread and butter of crow fauna I'm not gonna go too deep too into it but I'm gonna give you some important highlights so the graph panel is by default display ads blank with lines but you can also display it with the dots instead of lines if you find that useful useful I usually find this way too intrusive but in some cases it's much better because you can see it's easier to locate the
time stamp rather than with the lines you can also use bars and you if you use bars you might wanna stack them because otherwise the big bars will be locked in lower bars in some cases depending on the serine I'm override we also have a new feature for displaying the series on the x-axis instead of time so you can do bars on aggregate so the series can you can calculate the total of each area and display like this and whenever you create your dashboard I think it's important that you try to find unit to make it beautiful because that will make adoption much much likely and two of my favorite ways of doing this is the serious override so in this example which so shows percentiles I have changed the draw mode for the 95 percentile to use dots cause the 95 percentile usually is very volatile while the others percentiles are kind of steady so using dots for the 95 percentile makes the less inclusive best it's still there and the other one is if you have average zones of something you can also draw the mean and Max and feel the gap between them so you will see what the average is calculated by because sometimes Sturridge will hide information you depending on how big the gap is and this is a small and simple way of making sure you would catch that information as well and you can do that by displaying series overriding the display tab and then you specify a name of the serial or a reg X and then you can apply overrides from the before from the settings security having a panel this is something I'd love to do because nice panels is so much much much better another caution about the graph panel is when people use legends if you have mean max and average and a query containing aggregation like me then the mean will not be the wrong name it will be the mean of the averages and this is very common and confusing for because people are expecting to see the real mean value but that's never the case because graph on ax will only see the average value because they're I don't think there's any time series database that returns the average or me a max value in the query for the series as well that would be awesome but I don't think anyone lost the other panel that's very common people use is single stat and you can see use a few different settings for it the fourth one you can have a spark line to get some kind of sentence about the graph in the background it can have the Gorge and you can also map number value array number range to a string and these that might make it very easy to get get a feeling for about the state disease so in this case you can map it to unit codes so of course you should have smiled for your service status a bit caution about the support which for unicode is in your database prefer no support is but it's supported but if your database might not so check it before the third popular s panel is the table panel so this is a way to display time series tabular data and you can either this by a display down with a timestamp or you can do aggregation per serie and you can do coloring with thresholds and so on I think this also serves a good purpose in overview dashboard where you might want to get lost status of some Syria easy talk peace it down or something like that a very powerful feature in Ravana is the templating feature which allows you to add template variables which you can control from the dashboard and by modifying them you will modify all the panel area queries at the same time you can also use all and that kind of stuff to get to all the queries templating has a few different types first there's the custom one which allows you to predefined values that you can replace that you can learn later replacing the queries and you can also have queries that will send queries in this case to graphite asking for serve variables that you will find here and you can have template variables that depends on other template variables I can also have data sources as template variables in which case you have to specify which type of database and your phone will suggest the data sources you have this makes it easy to have one graph on our four different environments so you can have production and test and staging in from Bangor phone and reuse to that portion annotation is a way of highlighting events in the time series and by default they are added to all the graphs in the dashboard you can give them tags and you can also have links in the details so if you want to link to some systems with more information with about deploys or something are happening to the system it's really important to get that in the timeline as well I think a cool feature which was one of my first big features in Griffin is to play least one so you can create new playlists and then you can add dashboard to them either by name or you can use tags and then you say them and when you start a playlist you can either press one of these buttons or you can go to URL like this and since you can start a playlist from the URL it's very easy to set up a smaller computer like raspberry or something back on a TV put it in the lobby and then you can control the whatever dashboards are shown in the lobby from the from Griffin itself because every full cycle it will know the plane is from the data source again this is a bunch of shortcuts I don't think you see and one of them the most important one is if you stand over a panel and press E you go to edit mode
directly instead of having to click the title which is super nice for speed you can also share that port and if you have internal graph owner you might want to share it publicly and public profile instances means most of the case that anyone who can access the grifone instance can access your time save database and that's not a good idea because it's kind of easy to overload a time series database so instead of hosting Gravano publicly you should create snapshots MIDI data you wanna show gift lab had a maintenance problem last week and they did a really honorable way of publishing a lot of data kudos to them it was kind of sad to see the gravanos servers went bonkers in the same second a publicly published it using shell snapshots instead would have been no problem because when you publish them you can publish them to the grifone instance you're currently running or you can publish them to our graph owner and
we will publish them behind a CDN in the latest version of graphing I will also add a support for multi tool tip so if you name it enable it and hover Syria you get tooltip for every Syria and this is also very nice when you want to debug stuff in one I'll get a lot of data in the same dashboard and drill down and whenever you zoom or changes it will just follow it's a little bit it's a small performance impact so you might not want to have it on by default but it's only in your browser anyway so we've also recently added alerting and that was a it's been a really really really long standing issue in grifone on WE I think this is our third iteration on alerting and one of the first problem is that gravanos data sources is seen the front end so we create the queries in JavaScript with a lot of thing we can't do that because alerting has to run in background so we had a few different strategies for solving this and the first we tried was to execute your scripting they go back end and we said that we don't want to maintain this code it's possible but we don't wanna maintain it and it's complex and it's hurry and we don't wanna want our alerting code to be any way in any way complex so simplicity was really important for us which meant that we have to re-implement all data sources in the backend which limits plugins so plugins cannot have alerting right now we have the plans for adding plugins support in the backend in the future because gulang is going to release that soon but it's not ready yet but if you in in the graph panel you've got a new option for alerting and if you decide to create an alert you get all these options otherwise it's just about them asking for create an alert and the way it works is that you refer a refer to metric query in metrics tab you give it a time range five minutes to now and you give it a reducer and a reduced service a way to get a final number from a time serial so in this case we would make a calculate average of each each time series returned by this query and then we would compare it to this number and then you can slide the threshold of
around all that camera nifty stuff whenever an alert is triggered there's a few different alert notification channels slack of course which will snap take a screenshot of the graph where it's alerting it can give you all the Siri names that violated the threshold and some alerting your descriptions and
so on and this is the model for the alert so alert contains conditions and this can be an array of conditions and whenever one of these conditions are met well you can have war statements or an statements it depends on how you create your query but if this if the result of these somehow triggers we will send you an alert it might be simple compared to
all the solutions but we want it to be really simple our aim is not four hundred one hundred percent of all business cases we really want to satisfy
the beginner to medium complex users your phone at night as I mentioned earlier is our companion site we're gonna rename it Griffin Macomb eventually I the reasoning why we pick off on a net this strange even for us in hindsight where you can lease a hurricane you can find plugins both data sources and panels and you can also find the dashboard and a few panel a few plugins that I really like or the first one is the world map which allows you to map times there's data to he opened a new foreign communication or country codes which can be kind of nice to see there if you have multi data centers or you can see where customers are coming from or stuff like that the second favorite one is the diagram panel which allows you to draw diagrams like this and connect the boxes to a times area so it's it's not the weather map but it's something more simple and you create these diagrams by mark town not more time but DSL corner mid yes and the last one is the custom JSON data source which allows you to integrate any old legacy system to Ravana it's it's mainly a way for Reforma saying sending away request and expect your custom web app to respond with date with time series and the custom web app can also suggest queries and so on because there's always going to be some old system that you want to graph and you're never gonna just have one system for monitoring so this is a possibility to always keep moving and get everything into your phone that board was also introduced in trio but it's been growing steadily so if you work on a project like prometheus exporter or some framer work that exposes metrics you can create a dashboard upload it to Griffin Arnett and others will see that and can download the dashboard from graph on a net and this makes it much much much easier to get started with your product and it's it's a way of sharing and contributing to open source without having to code ingre fauna if you want to import a dashboard you just have to click import pattern you can enter a URL to graph on a net and it will import it
for you and then you have to select some properties like data sources alone and also said earlier sharing dashboards is a good good way of contributing back to the community and making sure everyone can consume and visualize metrics this is our first poster we created Ravana and yeah I think I think use the dashboard thing we added very much
alliance with this making everyone able to do matrix we also recently instead introduced host a graph on ax it's still in beta but if you're interested come poke me afterwards and I will hook up you just give the name and
the cloud will take care of the rest so that's enough about the present state let's talk about the future so the future contains four things that I want to mention I don't want to give a clear roadmap cause so far we have been changing our goals a lot and that's still our goal to do because we all work on the thing that currently is the most powerful so the first thing I know we're gonna do in the upcoming years iterate on alerting we're gonna add silencing we're gonna add the alert groups we don't support high availability right now there's no clear way of doing that that's something we wanna solve the second one is dashboard folders so many people started using organizations for teams that was never the intention of organizations organizations what's meant for company for hosting provider basically having different company using the same graph owner by introducing dashboards folders we hope to make it easier for people to have one graph on organization with different teams connected to one folder and folders will also include permissions and so on which all that kind of stuff that goes along with with arranging and structuring dashboard the third one is back-end plans right now we see a few different use cases first is alert notifications we currently support eight I think but there's no reason why we shouldn't be able to support endless amounts databases would also be great to have any back-end so people can add the custom databases and I'm alerting on them really simply in your problem authentication would also be a good fit for plugins and the fourth one is automation and that includes dashboard revision in some way of making sure that when this graph on our instance boots it has these data sources it has this dashboard it has this users such as etc we created the API for that use case but that's a stateful solution when a configurable solution instead not sure how to do that yet it's but we that's something we really really want to address without sacrificing startup speed but yeah that's about it for me I also have a bunch of graph on Oh t-shirts that I contact back with me because I don't shake language and if you contributed to grip honor I will give you a special bond so any questions I have one question if I want to implement your final solution say at my company we want to drive along a certain process runs every day and I am currently drowning in options with regards to time series database backends and so on what would I what would I need how should i what would be a quick start to implement such a demo so the first thing I would suggest is to just get started with one time series data base if you can't decide just pick one and get started and once you see the value you can pick the right one for you it's because I do agree there's so many right now it's hard to choose exactly but just choose one and you will get started and then you will see the value and then you can spend money decide a time and money deciding which one suits your use pays most No we support a lot of different data sources and all have what will return time series numbers so we will graph them all hi my question is a little bit strange I'm trying to integrate gravano with basic authentication basically hiding behind an internal server and somehow the basic authentication is not people and we can look at that awkward since that that's our kind of specific question I am if that's ok for you I'm leaving tomorrow and I stick around very simple question so we've seen that we can now import dashboards can you hear me can you hear me now yeah okay we've seen that we can import dashboard as JSON is it possible now to export them to Jason's without uploading them too yeah well you can be your yeaseong or you can export them to the format for importing so you can so you can export them and you get a verbal selection for data sources or so on yeah is that a plan to allow full-text search is there a plan to allow full text search like I would say in Cabana in Cabana there is a free text field and it would be really awesome to have that in Tirana is there a plan - yeah that's something we've been considering but it's it's cents panels are suppose to buy a condom by themself ingre fauna it's China tricky to introduce but it would be possible yeah you could use template variables for that you can use pipe whatever in the template file you can just type whatever but there's not that yeah we should have our template variable just thank you next question fiamma semen yes on the roadmap you didn't mention templating is there you hear me on the roadmap you didn't mention templating like sorry arrays as template variables and stuff is that something you're looking at or sorry I can't caption there so I'm not sure but you people we've been thinking about adding pupils you can have a hidden value and display value it's not for euros people not yet not yet now the road map is too wide for that feature any other questions regarding the last question I can show you how to do use customization for the data a custom JSON data source for that because you can any other questions poke me via email / Twitter some someway IRCA Griffin are for alerting alerting on multiple servers like templates in the backend and the trends of one server so if he would run on multiple servers behind proxy would it sorry so gravamen runs on multiple servers at the same time would it generate alerts from those multiple service or how do you keep them in sync every server would generate a lot you would have to disable alerting on you should only enable alerting on one server and that's the HR HR issue if
you wanna address somehow I wanted to use the team's the organization's for teams and the issue I run into is that data sources are not global that's per per organization so I made a patch to support local organizations and I was actually working on getting its proper circuit requests but you're probably not gonna merge it because of the folders which respects yeah we don't intend to do make organization sharing in any way I'm sorry wave another question over here so this your beta need to reside always somewhere in a time series database or can I build for example a dashboard where a value comes from a
REST API call at that moment look too much like a website or a Twitter account or whatever the way you would do that is to implement our custom application that returns time series that's the easiest one you could also create a plug-in in Ravana that reads from any API any other questions we have time this is really kind of nipping but is that is there any way to to get the latest feature to wait with displaying the new dashboard understand until it's fully rendered in the browser okay request that's a good feature with going once [Music] for annotations you said that by default or visible on all panels can we turn it off you can only enable it for a dashboard we added support in the code to have a pair panel but that's not configurable or ready for merchants so in alerting it would only be per panel see if you have alert for a graph panel and the alert triggers will create an annotation and that would only be rendered in that panel but so far we haven't added the like UX support for working with the annotations that way any more questions going once going twice : Thank You Carl so the next talk is in 25 minutes
Loading...
Feedback

Timings

  334 ms - page object

Version

AV-Portal 3.19.2 (70adb5fbc8bbcafb435210ef7d62ffee973cf172)
hidden