Grafana-As-Code: Fully reproducible Grafana dashboards with Grafonnet
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47342 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020463 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
CodeCodeComputer animation
00:26
INTEGRALComputer animation
01:13
Software maintenanceConsistencyGraph coloringError messageMultiplication signSet (mathematics)Revision controlFrequencyGraph (mathematics)Computer animation
01:53
Different (Kate Ryan album)CodeOperator (mathematics)Software developerComputer animation
02:20
Web browserDatabaseInternet service providerElectronic mailing listComputer fileInternet service providerDirection (geometry)CuboidDatabaseComputing platformWeb browserMultiplicationComputer animation
03:18
Variable (mathematics)Formal languagePhysical systemComputer animation
03:54
Presentation of a groupScripting languageComputer programmingVariable (mathematics)SubsetFormal languageResultantFunction (mathematics)Computer animation
05:15
Function (mathematics)Patch (Unix)Inheritance (object-oriented programming)Local ringLibrary (computing)Revision controlPoint (geometry)CollaborationismFormal languageComputer configurationDemo (music)Type theoryStatisticsFunctional (mathematics)Single-precision floating-point formatBitComputer fileScripting languageResultantKey (cryptography)Statement (computer science)MultiplicationSimilarity (geometry)Computer animation
09:14
Dependent and independent variablesComputer animation
09:35
Demo (music)Template (C++)File formatLocal ringGraph (mathematics)SpacetimeMiniDiscVertex (graph theory)Execution unitWhiteboardGoodness of fitQuery languageTemplate (C++)Boilerplate (text)Total S.A.Multiplication signInformationCASE <Informatik>MiniDiscMatrix (mathematics)Demo (music)SpacetimeComputer fileException handlingInstance (computer science)File systemDirectory serviceLine (geometry)Computer configurationDifferent (Kate Ryan album)Level (video gaming)Variable (mathematics)Library (computing)Interface (computing)Source codeParameter (computer programming)Perfect groupCodeNumberGraph (mathematics)Functional (mathematics)Execution unitComputer animation
15:19
outputGraph coloringFunction (mathematics)Graph (mathematics)SpacetimeMiniDiscComputer animation
15:59
SpacetimeCodeInformationMetric systemGraph coloringComputer animation
16:38
Execution unitMiniDiscSpacetimeVertex (graph theory)Local ringSource codeCodeOpen sourceSource codeQuicksortConfiguration spaceLibrary (computing)Projective planeFunctional (mathematics)Data conversionPower (physics)Link (knot theory)Computer animation
18:22
Disk read-and-write headMathematicsEvent horizonWeightMultiplication signVariable (mathematics)Beta functionRevision controlLibrary (computing)VolumenvisualisierungSoftware developerPlanningRight angleVirtual machineProcess (computing)CodeConfiguration spaceScripting languageTime seriesOpen setSelf-organizationMereologyCartesian coordinate systemHuman migrationSet (mathematics)Software repositoryStability theoryLaptopField (computer science)Computer animation
25:27
Point cloudFacebookOpen source
Transcript: English(auto-generated)
00:05
I'd just like to introduce Malcolm and Julien. He's going to talk about Grafana's code. So hello, I'm Malcolm Holmes from Grafana Labs and I am from Inuit.
00:27
Why would you need to talk today about Grafana dashboards and putting them into some kind of code, but the first question is why would you need to do that? Because in Grafana you have a rich UI, you can quickly make a lot of nice dashboards that are all looking
00:46
really nice and at the end it's so nice that you actually want more and more and more dashboards inside your Grafana, so you understand what's going on in your infrastructure, you understand a lot of stuff just by looking at all of those dashboards. And there are lots of new reasons to
01:02
get dashboards that are coming on like the integration of the logs, integration of new panels, it's really really quick to create new dashboards. And at the end what you have when you want to maintain your set of dashboards over a long time period is that you will have some conflicts
01:21
between people that prefer one way the other way or you will have a lot of confusion because in one graph you will show errors in one color and the other one in another color then there will be a new feature in Grafana and you need to change 50 dashboards to use that new feature and then you will become becoming a lot inside just working around all features, working on a
01:47
lot of stuff to get your dashboards looking good. So that's why you kind of want to use your dashboard as code and also it's very important that you can get when you have a dashboard to have everyone that will look the dashboard to read the same thing, to understand
02:03
the dashboard exactly like you understand it. And it makes that you want to reuse the same code, the same methodology, the same things in different dashboards so that when the developers see the dashboard and an operation guy sees the dashboard they see the same thing and that's very difficult to do. So what can we do about that? So you can provision
02:26
Grafana in multiple ways. So we have already talked about the browser UI. Grafana is a REST API since forever that enables you to create dashboards inside the UI and that goes then
02:40
into the database and you have a platform provider to do that but more than that nowadays Grafana supports file provisioning out of the box which means that you can say to Grafana hey please go into that directory, please go into that and look at all the JSON files that are there and just like we will load that inside dashboards and when you update the files there
03:05
Grafana will automatically pick them up and update your dashboards directly. So which is really nice when you want to code your dashboards and keep them aligned with what you have in your files. So Grafana dashboards, they are JSON. So it's a really long list of JSON for each panel,
03:29
for each stuff in the dashboard, each variable. I think it's in JSON so that's quite easy but JSON itself it's like no fun. So if you have tried to template JSON in any templating language
03:43
you will quickly see that it's no fun, that you cannot have the trailing comma, that you cannot, there is a lot of stuff that makes JSON really hard to template with traditional templating systems. So what can you do about that? So what can we do about this? So we've already had presentation yesterday in this room about
04:07
a language called JSONit that can also be used for deploying to Kubernetes. But one thing, what JSONit is, is a language, it's a superset of JSON. So you can use this as a language
04:24
that allows you to generate JSON. The end result of executing a JSONit script is JSON. So you can see a few things here, so for example you define a local variable and then that same local variable is referenced there. You get nice little syntactic sugar, for example not having to use
04:46
the quotes and so on. And here you have this colon colon syntax means that it's available, that hidden is available elsewhere in your program but doesn't end up in the resultant JSON output. So JSONit has many many language features that make it a pleasure to use, to work with,
05:07
to generate JSON and to collaborate with others in generating JSON. So let's have a look at some aspects of the language. So for example here we have a very, I mean this is very truncated
05:22
and simplified, but so you can see we're defining a new dashboard with the title and the UID. We've got a little if statement there, if UID is set then use UID as the key and the value UID that comes from there as the value. And if we execute then, so that
05:42
defines a thing called dashboard, that's defining a function called dashboard. If we execute that here like so, this is the JSON we end up with. So we've got a language here where we can define functions that are then available to us to represent, so you can see if you imagine
06:01
that this dashboard JSON was a lot longer, we can potentially encapsulate all of that in one very very simple command. So patches, so let's say we have this snippet, this definition of a dashboard here, and the problem is that if I go back, you'll see here, do you see that schema
06:26
version is hardwired? The problem is that we now want to create a dashboard using schema version 22 instead of version 20, but the dashboard function hasn't exposed that. So I'm a bit
06:40
stuck aren't I really, but JSON it is really neat like that, so all I do is I call my JSON my function and I append to it this snippet of JSON and it simply adds overwrite schema version like so. So and that is extremely powerful, allowing you to have a function that says this
07:05
is what we think you're going to need, oh but if there's stuff that we haven't got thought of, hey it's all right you can still extend it. Imports, so not only have you got can you create functions, if you see here for example we've got we've put this function here new, same thing
07:24
into a file called dashboard.libsonnet. Now here in our main.json it we load that dashboard file into a local variable called dashboard dashboard and then we notice that I've made a mistake here, that should say dashboard.new and then we add on our schema version 22 and we end up with exactly
07:45
the same end result JSON. So now we've composed, decomposed our script into multiple files, so now we may find now that we've got the ability to create a dashboard we could have a hundred of these scripts like this to create all of our many many resulting dashboards.
08:04
Yeah getting one going one step further than this JSON it has a tool called JSON builder which is a bit like vendoring in Golang if you're familiar with that. So it can go and collect a JSON library from GitHub or similar places so you can interact with others in public
08:28
about your dashboards and your resources to yeah and that makes collaboration much more effective. So mentioning that so here are some libraries that exist you know the prominent one graphonit
08:44
which we'll use here is a very simple library that provides you with the basics creating a dashboard, creating a panel and so on creating a single stat panel and so on. Yeah so there you go that is so you can see some of the some of the options that are present in
09:07
the graphonit library. So at this point we give you a quick demo so I'm going to do the typing
09:21
Julien's going to do the talking. Yeah but we will share the responsibility if that fails. No idea I can make it bigger yeah there you go okay good.
10:06
Perfect. Okay so we are in a demo directory when you can already see that we have prepared some stuff so
10:21
we have some gsonnet files that will declare our dependencies which in this case will be graphonit and now we have libsonnet file so a libsonnet file is the same as a gsonnet file except that it is meant to be imported and this is a libsonnet file that contains
10:42
that will contain the dashboard to show some disk space things so basically just the min and the file. So you see that at the beginning we are declaring some variable and importing graphonit itself and then we have some functions that will enable us to create a new dashboard so we add a few things like a template so you can decide which instance
11:07
you want to see in your dashboard and then we will list all the different file systems as you can see. So this is basically how you write a dashboard except that behind each one of those lines you have a lot more variables that are actually hidden by the library because when you
11:26
create a graphonit dashboard actually you need a lot of information like the time ranges that you can have like a lot of other stuff in this case we keep it quite simple and we only focus on the stuff that we want to see in the dashboard which in this case will be like
11:43
the the templates we want to use and then if we go below we go to the panels themselves so you see here that we will add a graph panel which will contain like five we want to override five different things in that graph panel and then we will add two queries to see basically
12:04
the total disk space and then the used disk space. So that is basically the boilerplate for our dashboard. As you can see we have some total matrix and fee matrix variables that we can have
12:22
so that we will be able to reuse that later on and to actually reuse the exact same dashboard to show two different things so when you will be able to read one dashboard will be able to read the second one. So now if we open the dashboards.json here we what we have is actually a map
12:43
of dashboards that will import the file that we have just seen so and it's basically a JSON map that will call the function that we have created just before and it will be created once to see the disk space as in bytes and the same dashboard will be used to show
13:02
the disk space but as indeed the number of inodes that are used and free. So the only differences that we have between the two dashboard would be the name of the matrix the title and the units that you want to see but the dashboard will be exactly the same and now we will just compile that and provide that to Grafana. So now we are running we are
13:34
launching a Grafana instance which at this stage is empty because we haven't compiled the dashboards yet. What we have not shown there is that we have also defined the dashboards as
13:47
code so this is a really fresh Grafana instance but I mean the data source is already there because we have configured it as code as well so we don't need to go to the interface to configure the the data source parameters. So now we will just compile the dashboards into
14:04
a directory that we have provided to Grafana so that it will they will show up in the in the UI. So like every good demo there is something that goes wrong and now it should be right. Okay so what we now want to do is to to generate the JSON files.
14:39
So the dash j command is there to tell you which to import some external
14:47
directories when you can now import the file from so in this case we will use the vendor directory where we have the Grafana library and the dash m option is there so that you can provide a directory to JSONet and all the entries of the map in the file will become a file
15:04
in the directory. So now we should see when Malcolm will run the command two files being created by JSONet and now we have a disk files and a disk files dashboard that have been created on a disk and if we now open Grafana we should see the two dashboards there. So now we
15:34
have a first dashboard that will show us the disk space so as you see we made a very simple dashboard
15:42
in the real world you will see some reusable dashboards like this but also like some reusable panels and reusable overrides so like the same colors everywhere that kind of things and now if you can read that graph you can actually read the other one which is the one about the inodes so which should look exactly the same yeah so it is exactly the same dashboard
16:05
and we did reuse 99 percent of the code between the two dashboards so if you design it once and you want multiple teams to reuse it even if you now start to put colors like I want to see the use space in red and the free space in green you can actually now use that with any
16:21
kind of metrics to show exactly the same thing and always in the same way so when someone comes into such a dashboard they see colors they recognize it and they directly know okay that's that kind of information and I know how to read that information and just finally to say that
16:42
is all of the code that ultimately was needed to say these are the dashboards we want which strikes me as pretty neat and compact okay so
17:02
just to say there's there you know we've certainly been having certainly at Grafana labs we've been having quite a few conversations about this because we believe there's a lot of power to this there's a lot of power available here and we've got lots of ideas as to how this could you know what if Grafana itself had native JSON it functionality so you don't run
17:25
the JSON it to make JSON to pass it there what if you just pass the JSON it up and it did it you know all sorts of interesting ideas there about how this could interact just basically make it really easy such that the you have your configuration for Prometheus or whatever other
17:43
data sources you have and you maintain that in in git like we all do but but so but this makes us able to take the dashboards as well and to treat those as code that travel in the same way as our yeah in the same way as the rest of our code so it is so Grafana it is an open source
18:09
project you can see the link there you know go try it I expect over the coming year or so that library could grow quite significantly yeah yeah so thank you any questions
18:32
just raise your hand oh I see one over there do you consider will this be a mostly Grafana
18:42
labs driven thing or do you want to put like most of the future development on the community do you imagine like evolving this as a very an official library so I guess I guess if we're integrating JSON it into Grafana that's going to be done by Grafana developers whether they're
19:03
in Grafana labs or not but I think partly what what we'd like to do is to is to try and cultivate a Jsonnet ecosystem you know what if you were to use Jsonnet to deploy your applications to Kubernetes and oh hang on a sec when I deployed the applications I
19:22
also deployed my configuration for Prometheus and I also deployed my dashboards and they all just went there and they just you know that's what we do ourselves right now at Grafana and that's what we'd like to make try and find ways to make available more widely so when I
19:41
started Grafana I was like if I don't do it someone else will do it so I went to the Grafana guys and I told them if you can provide me on the Grafana organization a repo then we can build something with the community so this came out of the community and I hope that some days we get also more involvement from both the community and Grafana but currently this came out of the
20:04
community but it is used by multiple people and even internally at Grafana they are using dappsource for some stuff so contributions are welcome from the community anyway so the aim really is to create a set of tools that can facilitate a wide-scale community adoption
20:24
head over here hi I am using extend variables quite a lot of with Jsonnet do you think if you include Jsonnet into Grafana how proceed with this that's way ahead of us
20:42
yes no idea you have to think about that when we do it if I'm allowed to ask two questions okay we'll start with the first one so kind of rude questions why should I use JSON net over Grafana lib over Grafana lib from beefworks you can describe your dashboards using Python
21:10
so why should I use Jsonnet or Grafana net so to be fair when I started Grafana just because I
21:23
didn't have a Python 3 enabled CI machine but basically I really like the freedom that I have with Jsonnet to just do it directly very close to the Json that Grafana will render at the end and I can easily override any variable directly in in Grafana and basically in Grafana you
21:43
it is really very easy to create some kind of libraries that you can use and panels that you can reuse and I find out that Jsonnet was because it was close to Json that it was really easy to go from what Grafana will render at the end and what I have in my code did you have
22:08
a second did you have a second I think I saw some something around here as well so and just a question from your Dima do I understand it correctly that there are plans to integrate
22:24
Jsonnet more into Grafana so I thought why not upload the definition and define the definition with your Grafana so for every new tag in my time series generate a dashboard within Grafana without I need to do this on my laptop computer and upload it are there plans to
22:44
execute scripts within Grafana so I think I think I would say that you know this is this is something we've we've found to be extremely effective for managing
23:01
you know for managing our dashboards at Grafana labs and then pushing them as Json to the to to Grafana I all I'm saying suggesting is that we're playing with the idea to see what would it be like if we did that what would that open up for us and that's just a very much an open exploration yeah so the Grafana team is currently collecting
23:23
ideas around this to see up to where they you want to go to where they should bring that idea to get a script inside Grafana just and it's like Grafana so if you have good ideas like this if you have ideas about all that stuff you can just go to the Grafana team and they will welcome all the ideas yes cool thank you I think we have time for one more
23:49
no oh yeah you can ask one definitely just I'm just curious the Json definition of dashboards is it are they stable or change they often with new Grafana releases
24:01
I mean is it like a stable API or something that is a fantastic question yes and Grafana is fantastic because when they update the Json in the in the dashboards they will always migrate nicely to the last revision so even when they change some fields
24:21
there is always a migration part when Grafana will hit the dashboard it will see oh this is schema version 16 I can bring that to 20 by doing this is this and it basically always works so some of my dashboard but it must the event change in the five zero beta yeah at the same time you know if there was a more clear definition of dashboard schema
24:44
rather than having a the Grafana library being manually curated it could be just automated and you just say oh there's a new version right run a process there's your Grafana library use that and and the the Grafana library then tracks the development of panels and so on and that
25:04
could be a very interesting development so yes that's something to do okay so thank you again and I'll see you back here in 15 minutes for the last talk of the day so thank you again