We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Grafana-As-Code: Fully reproducible Grafana dashboards with Grafonnet

00:00

Formal Metadata

Title
Grafana-As-Code: Fully reproducible Grafana dashboards with Grafonnet
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Grafana configuration can nowadays be fully done as code, which enables code review, code reuse, and in general better workflows when working with dashboards. This talk will present Grafonnet, a Jsonnet library to generate Grafana dashboards and some tips and tricks about how to use it efficiently and how to manage fully your grafana instances from code. We will also explore how Jsonnet and Grafonnet enable collaboration on dashboards, using Mixins and explain how to push dashboards to Grafana, either using Kubernetes, or direct to the Grafana API.
33
35
Thumbnail
23:38
52
Thumbnail
30:38
53
Thumbnail
16:18
65
71
Thumbnail
14:24
72
Thumbnail
18:02
75
Thumbnail
19:35
101
Thumbnail
12:59
106
123
Thumbnail
25:58
146
Thumbnail
47:36
157
Thumbnail
51:32
166
172
Thumbnail
22:49
182
Thumbnail
25:44
186
Thumbnail
40:18
190
195
225
Thumbnail
23:41
273
281
284
Thumbnail
09:08
285
289
Thumbnail
26:03
290
297
Thumbnail
19:29
328
Thumbnail
24:11
379
Thumbnail
20:10
385
Thumbnail
28:37
393
Thumbnail
09:10
430
438
CodeCodeComputer animation
INTEGRALComputer animation
Software maintenanceConsistencyGraph coloringError messageMultiplication signSet (mathematics)Revision controlFrequencyGraph (mathematics)Computer animation
Different (Kate Ryan album)CodeOperator (mathematics)Software developerComputer animation
Web browserDatabaseInternet service providerElectronic mailing listComputer fileInternet service providerDirection (geometry)CuboidDatabaseComputing platformWeb browserMultiplicationComputer animation
Variable (mathematics)Formal languagePhysical systemComputer animation
Presentation of a groupScripting languageComputer programmingVariable (mathematics)SubsetFormal languageResultantFunction (mathematics)Computer animation
Function (mathematics)Patch (Unix)Inheritance (object-oriented programming)Local ringLibrary (computing)Revision controlPoint (geometry)CollaborationismFormal languageComputer configurationDemo (music)Type theoryStatisticsFunctional (mathematics)Single-precision floating-point formatBitComputer fileScripting languageResultantKey (cryptography)Statement (computer science)MultiplicationSimilarity (geometry)Computer animation
Dependent and independent variablesComputer animation
Demo (music)Template (C++)File formatLocal ringGraph (mathematics)SpacetimeMiniDiscVertex (graph theory)Execution unitWhiteboardGoodness of fitQuery languageTemplate (C++)Boilerplate (text)Total S.A.Multiplication signInformationCASE <Informatik>MiniDiscMatrix (mathematics)Demo (music)SpacetimeComputer fileException handlingInstance (computer science)File systemDirectory serviceLine (geometry)Computer configurationDifferent (Kate Ryan album)Level (video gaming)Variable (mathematics)Library (computing)Interface (computing)Source codeParameter (computer programming)Perfect groupCodeNumberGraph (mathematics)Functional (mathematics)Execution unitComputer animation
outputGraph coloringFunction (mathematics)Graph (mathematics)SpacetimeMiniDiscComputer animation
SpacetimeCodeInformationMetric systemGraph coloringComputer animation
Execution unitMiniDiscSpacetimeVertex (graph theory)Local ringSource codeCodeOpen sourceSource codeQuicksortConfiguration spaceLibrary (computing)Projective planeFunctional (mathematics)Data conversionPower (physics)Link (knot theory)Computer animation
Disk read-and-write headMathematicsEvent horizonWeightMultiplication signVariable (mathematics)Beta functionRevision controlLibrary (computing)VolumenvisualisierungSoftware developerPlanningRight angleVirtual machineProcess (computing)CodeConfiguration spaceScripting languageTime seriesOpen setSelf-organizationMereologyCartesian coordinate systemHuman migrationSet (mathematics)Software repositoryStability theoryLaptopField (computer science)Computer animation
Point cloudFacebookOpen source
Transcript: English(auto-generated)
I'd just like to introduce Malcolm and Julien. He's going to talk about Grafana's code. So hello, I'm Malcolm Holmes from Grafana Labs and I am from Inuit.
Why would you need to talk today about Grafana dashboards and putting them into some kind of code, but the first question is why would you need to do that? Because in Grafana you have a rich UI, you can quickly make a lot of nice dashboards that are all looking
really nice and at the end it's so nice that you actually want more and more and more dashboards inside your Grafana, so you understand what's going on in your infrastructure, you understand a lot of stuff just by looking at all of those dashboards. And there are lots of new reasons to
get dashboards that are coming on like the integration of the logs, integration of new panels, it's really really quick to create new dashboards. And at the end what you have when you want to maintain your set of dashboards over a long time period is that you will have some conflicts
between people that prefer one way the other way or you will have a lot of confusion because in one graph you will show errors in one color and the other one in another color then there will be a new feature in Grafana and you need to change 50 dashboards to use that new feature and then you will become becoming a lot inside just working around all features, working on a
lot of stuff to get your dashboards looking good. So that's why you kind of want to use your dashboard as code and also it's very important that you can get when you have a dashboard to have everyone that will look the dashboard to read the same thing, to understand
the dashboard exactly like you understand it. And it makes that you want to reuse the same code, the same methodology, the same things in different dashboards so that when the developers see the dashboard and an operation guy sees the dashboard they see the same thing and that's very difficult to do. So what can we do about that? So you can provision
Grafana in multiple ways. So we have already talked about the browser UI. Grafana is a REST API since forever that enables you to create dashboards inside the UI and that goes then
into the database and you have a platform provider to do that but more than that nowadays Grafana supports file provisioning out of the box which means that you can say to Grafana hey please go into that directory, please go into that and look at all the JSON files that are there and just like we will load that inside dashboards and when you update the files there
Grafana will automatically pick them up and update your dashboards directly. So which is really nice when you want to code your dashboards and keep them aligned with what you have in your files. So Grafana dashboards, they are JSON. So it's a really long list of JSON for each panel,
for each stuff in the dashboard, each variable. I think it's in JSON so that's quite easy but JSON itself it's like no fun. So if you have tried to template JSON in any templating language
you will quickly see that it's no fun, that you cannot have the trailing comma, that you cannot, there is a lot of stuff that makes JSON really hard to template with traditional templating systems. So what can you do about that? So what can we do about this? So we've already had presentation yesterday in this room about
a language called JSONit that can also be used for deploying to Kubernetes. But one thing, what JSONit is, is a language, it's a superset of JSON. So you can use this as a language
that allows you to generate JSON. The end result of executing a JSONit script is JSON. So you can see a few things here, so for example you define a local variable and then that same local variable is referenced there. You get nice little syntactic sugar, for example not having to use
the quotes and so on. And here you have this colon colon syntax means that it's available, that hidden is available elsewhere in your program but doesn't end up in the resultant JSON output. So JSONit has many many language features that make it a pleasure to use, to work with,
to generate JSON and to collaborate with others in generating JSON. So let's have a look at some aspects of the language. So for example here we have a very, I mean this is very truncated
and simplified, but so you can see we're defining a new dashboard with the title and the UID. We've got a little if statement there, if UID is set then use UID as the key and the value UID that comes from there as the value. And if we execute then, so that
defines a thing called dashboard, that's defining a function called dashboard. If we execute that here like so, this is the JSON we end up with. So we've got a language here where we can define functions that are then available to us to represent, so you can see if you imagine
that this dashboard JSON was a lot longer, we can potentially encapsulate all of that in one very very simple command. So patches, so let's say we have this snippet, this definition of a dashboard here, and the problem is that if I go back, you'll see here, do you see that schema
version is hardwired? The problem is that we now want to create a dashboard using schema version 22 instead of version 20, but the dashboard function hasn't exposed that. So I'm a bit
stuck aren't I really, but JSON it is really neat like that, so all I do is I call my JSON my function and I append to it this snippet of JSON and it simply adds overwrite schema version like so. So and that is extremely powerful, allowing you to have a function that says this
is what we think you're going to need, oh but if there's stuff that we haven't got thought of, hey it's all right you can still extend it. Imports, so not only have you got can you create functions, if you see here for example we've got we've put this function here new, same thing
into a file called dashboard.libsonnet. Now here in our main.json it we load that dashboard file into a local variable called dashboard dashboard and then we notice that I've made a mistake here, that should say dashboard.new and then we add on our schema version 22 and we end up with exactly
the same end result JSON. So now we've composed, decomposed our script into multiple files, so now we may find now that we've got the ability to create a dashboard we could have a hundred of these scripts like this to create all of our many many resulting dashboards.
Yeah getting one going one step further than this JSON it has a tool called JSON builder which is a bit like vendoring in Golang if you're familiar with that. So it can go and collect a JSON library from GitHub or similar places so you can interact with others in public
about your dashboards and your resources to yeah and that makes collaboration much more effective. So mentioning that so here are some libraries that exist you know the prominent one graphonit
which we'll use here is a very simple library that provides you with the basics creating a dashboard, creating a panel and so on creating a single stat panel and so on. Yeah so there you go that is so you can see some of the some of the options that are present in
the graphonit library. So at this point we give you a quick demo so I'm going to do the typing
Julien's going to do the talking. Yeah but we will share the responsibility if that fails. No idea I can make it bigger yeah there you go okay good.
Perfect. Okay so we are in a demo directory when you can already see that we have prepared some stuff so
we have some gsonnet files that will declare our dependencies which in this case will be graphonit and now we have libsonnet file so a libsonnet file is the same as a gsonnet file except that it is meant to be imported and this is a libsonnet file that contains
that will contain the dashboard to show some disk space things so basically just the min and the file. So you see that at the beginning we are declaring some variable and importing graphonit itself and then we have some functions that will enable us to create a new dashboard so we add a few things like a template so you can decide which instance
you want to see in your dashboard and then we will list all the different file systems as you can see. So this is basically how you write a dashboard except that behind each one of those lines you have a lot more variables that are actually hidden by the library because when you
create a graphonit dashboard actually you need a lot of information like the time ranges that you can have like a lot of other stuff in this case we keep it quite simple and we only focus on the stuff that we want to see in the dashboard which in this case will be like
the the templates we want to use and then if we go below we go to the panels themselves so you see here that we will add a graph panel which will contain like five we want to override five different things in that graph panel and then we will add two queries to see basically
the total disk space and then the used disk space. So that is basically the boilerplate for our dashboard. As you can see we have some total matrix and fee matrix variables that we can have
so that we will be able to reuse that later on and to actually reuse the exact same dashboard to show two different things so when you will be able to read one dashboard will be able to read the second one. So now if we open the dashboards.json here we what we have is actually a map
of dashboards that will import the file that we have just seen so and it's basically a JSON map that will call the function that we have created just before and it will be created once to see the disk space as in bytes and the same dashboard will be used to show
the disk space but as indeed the number of inodes that are used and free. So the only differences that we have between the two dashboard would be the name of the matrix the title and the units that you want to see but the dashboard will be exactly the same and now we will just compile that and provide that to Grafana. So now we are running we are
launching a Grafana instance which at this stage is empty because we haven't compiled the dashboards yet. What we have not shown there is that we have also defined the dashboards as
code so this is a really fresh Grafana instance but I mean the data source is already there because we have configured it as code as well so we don't need to go to the interface to configure the the data source parameters. So now we will just compile the dashboards into
a directory that we have provided to Grafana so that it will they will show up in the in the UI. So like every good demo there is something that goes wrong and now it should be right. Okay so what we now want to do is to to generate the JSON files.
So the dash j command is there to tell you which to import some external
directories when you can now import the file from so in this case we will use the vendor directory where we have the Grafana library and the dash m option is there so that you can provide a directory to JSONet and all the entries of the map in the file will become a file
in the directory. So now we should see when Malcolm will run the command two files being created by JSONet and now we have a disk files and a disk files dashboard that have been created on a disk and if we now open Grafana we should see the two dashboards there. So now we
have a first dashboard that will show us the disk space so as you see we made a very simple dashboard
in the real world you will see some reusable dashboards like this but also like some reusable panels and reusable overrides so like the same colors everywhere that kind of things and now if you can read that graph you can actually read the other one which is the one about the inodes so which should look exactly the same yeah so it is exactly the same dashboard
and we did reuse 99 percent of the code between the two dashboards so if you design it once and you want multiple teams to reuse it even if you now start to put colors like I want to see the use space in red and the free space in green you can actually now use that with any
kind of metrics to show exactly the same thing and always in the same way so when someone comes into such a dashboard they see colors they recognize it and they directly know okay that's that kind of information and I know how to read that information and just finally to say that
is all of the code that ultimately was needed to say these are the dashboards we want which strikes me as pretty neat and compact okay so
just to say there's there you know we've certainly been having certainly at Grafana labs we've been having quite a few conversations about this because we believe there's a lot of power to this there's a lot of power available here and we've got lots of ideas as to how this could you know what if Grafana itself had native JSON it functionality so you don't run
the JSON it to make JSON to pass it there what if you just pass the JSON it up and it did it you know all sorts of interesting ideas there about how this could interact just basically make it really easy such that the you have your configuration for Prometheus or whatever other
data sources you have and you maintain that in in git like we all do but but so but this makes us able to take the dashboards as well and to treat those as code that travel in the same way as our yeah in the same way as the rest of our code so it is so Grafana it is an open source
project you can see the link there you know go try it I expect over the coming year or so that library could grow quite significantly yeah yeah so thank you any questions
just raise your hand oh I see one over there do you consider will this be a mostly Grafana
labs driven thing or do you want to put like most of the future development on the community do you imagine like evolving this as a very an official library so I guess I guess if we're integrating JSON it into Grafana that's going to be done by Grafana developers whether they're
in Grafana labs or not but I think partly what what we'd like to do is to is to try and cultivate a Jsonnet ecosystem you know what if you were to use Jsonnet to deploy your applications to Kubernetes and oh hang on a sec when I deployed the applications I
also deployed my configuration for Prometheus and I also deployed my dashboards and they all just went there and they just you know that's what we do ourselves right now at Grafana and that's what we'd like to make try and find ways to make available more widely so when I
started Grafana I was like if I don't do it someone else will do it so I went to the Grafana guys and I told them if you can provide me on the Grafana organization a repo then we can build something with the community so this came out of the community and I hope that some days we get also more involvement from both the community and Grafana but currently this came out of the
community but it is used by multiple people and even internally at Grafana they are using dappsource for some stuff so contributions are welcome from the community anyway so the aim really is to create a set of tools that can facilitate a wide-scale community adoption
head over here hi I am using extend variables quite a lot of with Jsonnet do you think if you include Jsonnet into Grafana how proceed with this that's way ahead of us
yes no idea you have to think about that when we do it if I'm allowed to ask two questions okay we'll start with the first one so kind of rude questions why should I use JSON net over Grafana lib over Grafana lib from beefworks you can describe your dashboards using Python
so why should I use Jsonnet or Grafana net so to be fair when I started Grafana just because I
didn't have a Python 3 enabled CI machine but basically I really like the freedom that I have with Jsonnet to just do it directly very close to the Json that Grafana will render at the end and I can easily override any variable directly in in Grafana and basically in Grafana you
it is really very easy to create some kind of libraries that you can use and panels that you can reuse and I find out that Jsonnet was because it was close to Json that it was really easy to go from what Grafana will render at the end and what I have in my code did you have
a second did you have a second I think I saw some something around here as well so and just a question from your Dima do I understand it correctly that there are plans to integrate
Jsonnet more into Grafana so I thought why not upload the definition and define the definition with your Grafana so for every new tag in my time series generate a dashboard within Grafana without I need to do this on my laptop computer and upload it are there plans to
execute scripts within Grafana so I think I think I would say that you know this is this is something we've we've found to be extremely effective for managing
you know for managing our dashboards at Grafana labs and then pushing them as Json to the to to Grafana I all I'm saying suggesting is that we're playing with the idea to see what would it be like if we did that what would that open up for us and that's just a very much an open exploration yeah so the Grafana team is currently collecting
ideas around this to see up to where they you want to go to where they should bring that idea to get a script inside Grafana just and it's like Grafana so if you have good ideas like this if you have ideas about all that stuff you can just go to the Grafana team and they will welcome all the ideas yes cool thank you I think we have time for one more
no oh yeah you can ask one definitely just I'm just curious the Json definition of dashboards is it are they stable or change they often with new Grafana releases
I mean is it like a stable API or something that is a fantastic question yes and Grafana is fantastic because when they update the Json in the in the dashboards they will always migrate nicely to the last revision so even when they change some fields
there is always a migration part when Grafana will hit the dashboard it will see oh this is schema version 16 I can bring that to 20 by doing this is this and it basically always works so some of my dashboard but it must the event change in the five zero beta yeah at the same time you know if there was a more clear definition of dashboard schema
rather than having a the Grafana library being manually curated it could be just automated and you just say oh there's a new version right run a process there's your Grafana library use that and and the the Grafana library then tracks the development of panels and so on and that
could be a very interesting development so yes that's something to do okay so thank you again and I'll see you back here in 15 minutes for the last talk of the day so thank you again