gstat_exporter: Graphing FreeBSD disk utilization with Prometheus
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46997 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202059 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Programmable read-only memoryGraph (mathematics)MathematicsMultiplication signFigurate numberNeuroinformatikLecture/Conference
00:44
Graph (mathematics)MiniDiscTouchscreenDemo (music)Software developerPhysical systemInformation securityHacker (term)LaptopTwitterCopenhagen interpretationOperator (mathematics)Convex hullMIDIExecution unitMiniDiscUtility softwareBlogMetric systemService (economics)LaptopWeb browserDifferent (Kate Ryan album)Projective planeVirtual machineBitComputer programmingBuffer solutionBit rateTexture mappingMultiplication signSystem administratorPercolation theoryCopenhagen interpretationPhysical systemConnected spaceExterior algebraPoint (geometry)Moment (mathematics)Condition numberLevel (video gaming)Open sourceDemo (music)Server (computing)RecursionDirect numerical simulationIn-System-ProgrammierungHacker (term)Self-organizationContext awarenessInternetworkingBuildingSinc functionComputer animation
03:57
Series (mathematics)DatabasePhysical systemQuery languageData modelSource codeWritingPoint cloudControl flowMultiplication signPercolation theoryTwitterInformationTime seriesData managementPhysical systemState observerDatabaseVideo gameException handlingDimensional analysisSoftware testingSinc functionWordMetric systemSet (mathematics)Graph (mathematics)Point cloudCodeOpen sourceBitNumberConfiguration spaceFlow separationWeb 2.0File systemProcess (computing)Query languageComputer fileComputer animation
06:48
EmailPhysical systemWritingPoint cloudSeries (mathematics)Control flowVisualization (computer graphics)Gateway (telecommunications)MiniDiscSlide ruleGraph (mathematics)Service (economics)Metric systemCellular automatonInformation retrievalServer (computing)Projective planeRegular graphSoftwareIsing-ModellOpen sourceService (economics)Prisoner's dilemmaCentralizer and normalizerComplex (psychology)Metric systemElectronic mailing listData managementBlack boxEndliche ModelltheorieThresholding (image processing)Identity managementPhysical systemIP addressConnectivity (graph theory)Query languageComputer fileProcess (computing)Cloud computingFormal languageCuboidMultiplication signWebsiteLine (geometry)Key (cryptography)DiagramConnected spaceServer (computing)Data storage deviceIn-System-ProgrammierungTime seriesMiniDiscQuicksortExtension (kinesiology)DatabaseLecture/ConferenceComputer animationProgram flowchart
09:57
MiniDiscSlide ruleGraph (mathematics)Client (computing)Data modelMetric systemDisintegrationBlack boxCellular automatonGame theoryInternetworkingPhysical systemGraph (mathematics)Goodness of fitMetric systemCuboidShared memoryProjective plane2 (number)Direction (geometry)Computer animationLecture/Conference
10:45
Graph (mathematics)MiniDiscSlide ruleClient (computing)Data modelMetric systemDisintegrationQueue (abstract data type)BitMiniDiscComputer hardwareGraph (mathematics)Metric systemMessage passingClient (computing)Router (computing)Formal languagePhysical systemSingle-precision floating-point formatCodeLine (geometry)Library (computing)LengthServer (computing)Black boxMathematicsMereologyWeb 2.0Presentation of a groupMachine visionSymbol tableEntropie <Informationstheorie>Point (geometry)Computer animation
13:03
Point (geometry)MiniDiscMultiplication signWorkstation <Musikinstrument>Beta functionComputer clusterHigh availability2 (number)NumberData managementWeb 2.0Operator (mathematics)Term (mathematics)Server (computing)Front and back endsMetric systemSoftwareInstance (computer science)Decision theoryDatabaseService (economics)Archaeological field surveyGoodness of fitWhiteboardInheritance (object-oriented programming)Lecture/ConferenceComputer animation
15:14
MiniDiscGraph (mathematics)Slide ruleRange (statistics)Vector graphicsGraph (mathematics)Vector spaceCodierung <Programmierung>Series (mathematics)Metric systemVideo game consoleRegulärer Ausdruck <Textverarbeitung>Server (computing)Cheat <Computerspiel>Key (cryptography)View (database)AuthenticationComputer networkWeb pageMathematicsView (database)Formal languageStatisticsTotal S.A.Query languageMetric systemInternetworkingConnected spaceComputer animation
15:49
Slide ruleMiniDiscGraph (mathematics)Metric systemSeries (mathematics)Range (statistics)Codierung <Programmierung>Regulärer Ausdruck <Textverarbeitung>Vector spaceVector graphicsGraph (mathematics)Video game consoleServer (computing)Cheat <Computerspiel>Labour Party (Malta)Matching (graph theory)Process (computing)ExpressionFuzzy logicMultilaterationNumberSerial portGraph (mathematics)Total S.A.Vector spaceRegulärer Ausdruck <Textverarbeitung>Poisson-KlammerRange (statistics)Metric systemServer (computing)Computer animation
16:45
Bit rateQuery languageSummierbarkeitCheat <Computerspiel>Slide ruleMiniDiscGraph (mathematics)Resource allocationSpacetimeMetric systemVirtual machineSoftwareTraffic reportingNumberType theoryMetric systemBeta functionMiniDiscWordPatch (Unix)Virtual machineComputer configurationInstance (computer science)BitForcing (mathematics)Web pageGame theorySemiconductor memoryElectronic mailing listGraph (mathematics)QuicksortPhysical systemSoftwareBit rateUtility softwareNormal (geometry)Server (computing)Operator (mathematics)DatabaseBand matrixComputer animation
19:10
Programmable read-only memoryPermianSlide ruleMiniDiscOvalNumberRational numberWeightOperator (mathematics)Default (computer science)WritingReading (process)GeometryLecture/Conference
19:45
Slide ruleMiniDiscNumberFlagGraph (mathematics)Observational studyBasis <Mathematik>Electronic visual displayDifferent (Kate Ryan album)Web pageFunction (mathematics)Reading (process)BitPosition operatorAsynchronous Transfer ModeComputer animation
21:00
Slide ruleMiniDiscGraph (mathematics)Library (computing)Client (computing)Line (geometry)Source codeCodeGauge theoryService (economics)Web pageDifferent (Kate Ryan album)2 (number)FrequencyElectronic visual displayTrailKernel (computing)Computer fileBitWikiWebsiteInformationLine (geometry)Library (computing)CodeClient (computing)FamilyExpert systemRight angleFluid staticsComputer animation
22:39
Slide ruleMiniDiscGraph (mathematics)Dependent and independent variablesMiniDiscMetric systemSpacetimeWordFunction (mathematics)NumberMultiplication signCombinational logicGraph (mathematics)IdentifiabilityInformationLine (geometry)GeometryComputer animation
23:53
INTEGRALService (economics)Source codeMiniDiscLabour Party (Malta)Server (computing)Instance (computer science)Arithmetic meanLecture/ConferenceComputer animation
24:40
Graph coloringConnected spaceInternetworkingArithmetic progression2 (number)Address spaceDesktop publishingGraph (mathematics)Lecture/Conference
25:18
Demo (music)Hill differential equationDynamic random-access memoryPhysicsCommunications protocolPasswordError messageInstallation artProcess (computing)Uniform resource nameAddress spaceView (database)CloningProgrammable read-only memoryLaw of large numbersRevision controlLibrary (computing)Computer animationSource code
25:57
Revision controlView (database)PasswordInterior (topology)BackupPublic key certificateOnline helpGroup actionRootDrum memoryMach's principleMenu (computing)Local GroupHolographic data storageComputer fileIntrusion detection systemMessage passingHash functionAuthenticationShared memoryTerm (mathematics)Computer animationSource code
26:32
Chi-squared distributionConnected spacePasswordShared memoryView (database)Communications protocolLaw of large numbersHill differential equationService (economics)Revision controlDrum memoryComputer fileEvolutionarily stable strategySynchronizationProgrammable read-only memoryMaxima and minimaComputer animationSource code
27:15
Menu (computing)Connected spaceShared memoryHash functionInflection pointView (database)Communications protocolAddress spaceHill differential equationMaß <Mathematik>MIDIElectronic meeting systemComa BerenicesExecution unitLaw of large numbersInterior (topology)Term (mathematics)Computer fileLibrary (computing)Error messageControl flowAuthenticationProcess (computing)PasswordRevision controlPolygon meshDisk read-and-write headEvolutionarily stable strategyComputer wormNormed vector spaceProgrammable read-only memoryElectronic data interchangeInstance (computer science)Drum memoryInsertion lossMultiplication signSource code
28:09
Convex hullInclusion mapShared memoryLaw of large numbersHill differential equationProcess (computing)Computer fileView (database)PasswordError messageHidden Markov modelSolitary confinementAnnulus (mathematics)Address spaceInflection pointControl flowElectronic data interchangeHash functionMIDICommunications protocolGeometryFlagMiniDiscSlide ruleBitSinc functionDemo (music)Multiplication signAttribute grammarCodeFlagProjective planeBlogLine (geometry)Graph (mathematics)Source codeLecture/ConferenceComputer animation
29:58
Slide ruleFlagGeometryPhysicsMenu (computing)Chemical equationMetric systemTable (information)Computer animationSource code
30:35
Filter <Stochastik>Graph (mathematics)Different (Kate Ryan album)Band matrixNumberSheaf (mathematics)Set (mathematics)MiniDiscGeometryQueue (abstract data type)Server (computing)Metric systemForcing (mathematics)Labour Party (Malta)Associative propertyWebsiteSerial portRoutingRight angleMusical ensembleFamilyComputer animation
32:08
GeometryPattern languageGraph (mathematics)String (computer science)MiniDiscServer (computing)Process (computing)NumberHypermediaMultiplication signAreaService (economics)Motion captureRight angleComputer animation
33:38
Execution unit2 (number)Associative propertySet (mathematics)Revision controlSanitary sewerScaling (geometry)NumberSemiconductor memoryCommutatorData storage deviceAttribute grammarService (economics)Metric systemMiniDiscPhysical systemRight angleComputer animation
35:00
Identity managementServer (computing)MathematicsEndliche ModelltheorieData storage deviceMetric systemGraph (mathematics)CodeCentralizer and normalizerException handlingHacker (term)Query languageWordBitDependent and independent variablesFigurate numberData managementPoint (geometry)Instance (computer science)CountingConnected spaceService (economics)Different (Kate Ryan album)Open sourceCausalityArithmetic progressionProjective planeMultiplication signPosition operatorDesign by contractNumberShared memoryForcing (mathematics)Observational studyNatural languageEvent horizonComputer animation
41:12
Semiconductor memoryMultiplication signMiniDiscShared memoryGraph (mathematics)Lecture/Conference
42:01
Server (computing)StatisticsMultiplication signArithmetic progressionShared memorySemiconductor memoryServer (computing)AreaTraffic reportingSpacetimeChief information officerSquare numberVulnerability (computing)WeightRemote procedure callPoint (geometry)MetreNumberSinc functionGraph (mathematics)Source code
44:04
Semiconductor memoryMultiplication signPoint (geometry)Electronic mailing listEmailMiniDiscNumberSingle-precision floating-point formatInheritance (object-oriented programming)Open setReflection (mathematics)Arithmetic meanUltraviolet photoelectron spectroscopyMetropolitan area networkLecture/Conference
45:21
Point cloudOpen source
Transcript: English(auto-generated)
00:07
Thank you. Hello everybody. We have plenty of time. I submitted this as a lightning talk and it was allocated in an hour. So if you have questions or comments or anything
00:22
please feel free to interrupt me. Does anybody use Prometheus already? Or is this new for everybody? A few people? Okay, have you heard of it? Great. We got a Prometheus exporter in FreeBSD 12, so somebody in the FreeBSD world at least knows and uses Prometheus
00:45
apart from me. So this was a few months ago. I wrote an exporter to make disk utilization metrics available to Prometheus from FreeBSD. And I was encouraged to come here and talk
01:04
about it. I did a blog post about it and was encouraged to come here and tell you. So since many people probably don't know Prometheus, I'll start out by talking a little bit about what it is and what it does and why it's awesome. And then the concept
01:23
of exporters, and then we'll get to Gstat and the actual exporter that I wrote for that. I also have a Grafana dashboard to actually visualize the data collected. And if everything works and my internet connection keeps working, then I have a live demo of some servers at
01:43
my workplace using this. So it's always nice to see some actual real data. But I'm not making any promises because my connection has been flaky. So let's see when we get to it. Live demos are always, as you know, a bit tricky sometimes. So my name is Thomas Rasmussen. I'm called Tuckling on the internet. I'm born and live
02:09
in Copenhagen, in Denmark. And I do system architecture, build and administer FreeBSD servers. I do some programming as well, and was recently laid off with three months with
02:27
salary. So I'm just going to Fustum and relaxing and enjoying myself at the moment and looking for something else to do. I run a DNS service called Uncensored DNS for 10 years now, 11 years, which is before Google DNS and before all the other public
02:44
recursive DNS server services. I started this as an alternative to the ISP-sensored, often censored ISP DNS servers. And it's been running since. I also help organize Bon Hack, which is an annual Danish hacker camp. I highly recommend going. It is a great
03:03
time. It's a week with tents and laptops and hacking around with all kinds of fun things. And I have a bunch of minor open source projects on GitHub. I use FreeBSD exclusively on servers. I've used this since 5.2.1, I think, was my first installed server.
03:25
And on my laptops, I run something called QubesOS, which is an awesome rating system. I wish it was FreeBSD-based, but it's Fedora and Zend-based. The concept is you have different virtual machines for different contexts. So I have one for work and one for playing with
03:43
the Bon Hack stuff. And they can really disposable browsers for surfing. So the VM just disappears. So if you happen to click something bad, then just nothing really happens. Awesome concept. So I'm here to talk about this G-STAT exporter.
04:03
This is actually the first time I've been in the BSD room. There's never been room, but one of the perks of being a speaker is that you have to let me in. So I wrote this tweet, and someone picked up on it and recommended I come to FASTERM and talk about it, and here we are. So Prometheus, in their own words,
04:26
it's an open source monitoring system, dimensional data model, flexible query language, efficient time series database. And that is pretty much it. The whole ecosystem around it is very Unix-y in that they do one thing and they do it well. Prometheus in itself is
04:44
a time series database. It really doesn't do anything except collect metrics and stick them in a time series database. And then it has an API to make them available to Grafana, for example, so you can pull the data back out. It doesn't do alerting. There's something called Alert Manager, which from the same team,
05:04
but it's a separate piece of software, separate configuration file and stuff, which handles alerting and alert dependencies and which team should get what alert and pager duty support and all that stuff.
05:21
Monitoring has often, I think, in many years, it's been a bit of a, like, there was a void left when Nagios kind of stopped being modern, and I've been using Sarbix and a few other things, but nothing has really been, it's like they tried to do too much somehow. It's a big job
05:40
both to gather the data and visualize it and do alerting and everything. And I haven't really been happy with anything. And I think, you know, like the first time we tried CFS and thought this is what a file system should have been like all along. Where have you been all my life? This is how I feel about monitoring with Prometheus. It is an excellent system. So it is based on, they call it dimensional time series data.
06:05
It means that your metrics can have labels, they call it. So if you have a metric called HTTP requests total, for example, for a web server, and you get a number when you scrape the data, say you had 100 requests, then that data can have multiple dimensions
06:24
in that it can have labels to say what the path of the request was or what the HTTP status code was. So you can ask to get only the HTTP 404 requests, for example, and then make a graph of those in your Grafana. This is incredibly powerful. It's actually a
06:42
relatively simple concept, but it enriches the data greatly. Prometheus and Alert Manager and all the accompanying software is all written in Go. It was started at SoundCloud some years ago. It's been open source since 2015 and there's no company that owns it now. It's a
07:05
regular open source project with many contributors and a great community, a great ISE channel and stuff. So if you have questions, they're very happy to help. So this is the diagram of the ecosystem from their website. Central to the whole thing is the Prometheus
07:22
server. It's a time series database and it stores the metrics, of course, on some sort of disk storage. It's a pull model Prometheus. That's one of the things that some people have to get used to, which means that Prometheus connects to the stuff it's monitoring and pulls the
07:41
metrics over HTTP. There's various ways around that if you absolutely cannot do it like that, but that is the idea. For example, the FreeBSD SysHL exporter that came and arrived in FreeBSD 12 is supposed to be run under INETD and Prometheus connects to it and pulls out the metrics.
08:03
So that's what the exporters make available. They open an HTTP endpoint and it's just a plain text key value list of lines. I'll have examples later. There's something called PushGateway that if you have a job that runs for an hour and then
08:21
has some metrics to submit at the end of that, you can use PushGateway for that. It has a very extensive service discovery system, so if you are an AWS Kubernetes user or you use one of the many, many cloud services available, there's probably a service discovery thing available for it. At work, I worked in an ISP until I got laid off. We used the file
08:47
service discovery thing and just exported a list of customers and IP addresses and that was automatically ingested by Prometheus. So when we add a new customer in our central provisioning system, it automatically shows up and starts pulling the data from it.
09:04
And to talk to Prometheus and query the data and get it out, they invented something called PromQL. It doesn't really have anything to do with SQL. It is a custom query language
09:22
specifically for this and there will be examples of this later. It is a very powerful and easy to use language. And like I said, AlertManager is a separate component, so when you add some threshold or something you want to be alerted on in Prometheus, it pushes an alert to AlertManager
09:43
and AlertManager handles all the nitty-gritty with who is on vacation now and who has the
10:01
internet. We generally speak of blackbox and whitebox monitoring. Blackbox monitoring is when you ping your whatever or just poke something from the outside, check if it's listening on port 80 or whatever you might do. Whitebox monitoring is when the thing being monitored is kind of in the game. So for example, the FreeBSD sister cell exporter is a good example or an exporter
10:29
for gstat data, for example. It means you're kind of inside the system exposing the metrics to the outside and that's a much better way to do it than blackbox monitoring. It has support for
10:42
that as well. If you need to, like we do at work, ping 10,000 customers every five seconds, then you do that with something called blackbox exporter. But mostly and primarily it is whitebox monitoring. It is supposed to be used with the exporters built into whatever
11:01
you are monitoring. So if you have a system like our provisioning system at work, it's easy to, for example, export metrics for the queue length for provisioning the customers. The router needs to be, the switch ports need to be configured and there's sometimes a queue if there's
11:21
any changes at once. And it's very easy when you're in the code to export this metric and just do a slash permit use or slash metrics endpoint in your system and export it. It's over HTTP, like I said, and it is designed to handle everything you can throw at it.
11:44
You shouldn't be picky. You should instrument all parts of your system. Anything that might someday make sense and even if you think it doesn't because sometimes these subtle little changes in whatever can turn out to be important and then it's very nice to have the history. It is very efficient. It is designed to handle hundreds of thousands of metrics on a single
12:08
server commodity hardware. It has a little bit of, when you're exploring the data in the web interface, it can draw a simple crude graph, but it's not meant for graphing. It's meant to
12:22
be used with Grafana or something else, but I highly recommend. It is excellent integrated into Grafana. There's client libraries in many languages. The Gstatics code I wrote is written in Python and you start out with these 10 lines or something from the Prometheus example client and you just add name, define your metrics, saying we have a, in Gstat, we have a disk IO
12:46
busy present message metric or whatever and you feed data into them and it takes care of listening on HTTP and serving the data up in the right order. It is easy to use and they
13:00
have, of course, in Go and many other languages, client libraries, so it is easy to get started writing your own exporter. So it is, like I said, very efficient. They say about three and a half bytes per data point and it is remarkably efficient when pulling in data.
13:21
We went to, we replaced our Sabix installation, which was thrashing the disks using Postgres as a backend. I love Postgres. I don't love Sabix, but it was a decent setup, but it was, I mean, we could barely handle pulling them once per minute, the five, eight thousand customers,
13:42
however much we had back then. And when we switched to Prometheus, we're pulling them once every five seconds. We're doing additional metrics besides what we did with Sabix and the server is bored now. It's not doing nothing. So we went from absolutely thrashing the drives to the server being almost idle. It is amazingly efficient. If you go beyond what one server
14:05
can handle, it supports sharding and federation, so you can have many Prometheus servers working together and it does that very well. And alert manager as well supports high availability stuff, so you can send an alert to any number of alert manager instances and only one of them will
14:21
actually alert you. At work, the Prometheus installation I was managing there had about several hundred and fifty thousand time series and about thirteen and a half thousand per second metrics put into the database, and that's small in Prometheus terms. That's not an issue
14:44
at all. You can do that on a laptop or on whatever. It's in a jail, all my stuff, buns and jails, and you can easily have one for each team. It's not like managing an ELK instance or something like that where you have to, you can just web up a new if you want one for
15:02
the network team and one for the operations team or whatever, you can easily do that. Okay, so the query language from QL. I brought a few examples and it's easy to look up. It has
15:27
a page or two and then if you understand basic statistics and math, then it should be easy to use. First things first, HTTP request total is the example metric we're using here.
15:44
So that was the internet connection that stopped working. So that just returns the metric HTTP request total. If, for example, I want the metric but only
16:00
if the job label is API server and the handler is slash API slash comments, then I put it in the curly brackets and it only returns the number of HTTP requests that had those labels. You can also return a range vector so you get all the values for a given, for example,
16:25
for a five-minute interval, if you need that. You can use regular expressions for the labels and you can do first-year matching and stuff for the label values as well. I use those labels a lot in the gstat exporter to save stuff like the disks,
16:43
serial numbers and stuff. We'll get to that later. A bit more advanced, it has some keywords to do. For example, it can calculate the rate over five minutes for this metric or even do a quantile 95th percentile is used a lot when
17:05
when charging for bandwidth and hosting operations and stuff like that or wherever you need to to use it. It is very flexible. It can also show, you can query the number of alerts firing
17:22
in a system and when the installation grows, that's the sort of stuff you want on your front page of your Grafana instance. You can have zero or five alerts or whatever. Prometheus connects to an exporter and gets fat metrics and ingests them.
17:44
And these metrics are exported by exporters. Machine metrics like the normal metrics like RAM and an IO and network traffic and stuff like that are exported by something called node exporter, which we have in ports and it works well. But on FreeBSD, it doesn't have
18:06
disk utilization metrics. It does on Linux, of course, but not on FreeBSD. Which is why I started out because when you're troubleshooting stuff on a database server or something, it is very nice to have metrics for disk utilization. There's exporters for
18:29
every type of software you can almost imagine. There's hundreds of them. Each is allocated a port on this list. You just go to the wiki and allocate a port and for the gstat exporter, I got port 92 something. But that's a good list and you can search for
18:48
FreeBSD on it and find a few others. There's a jail exporter, which is very nice, that somebody made that uses RCTL to get IOPS and memory use and stuff for jails and then exports
19:00
it in a way so Prometheus can ingest it. So you can get graphs to say how much RAM each jail is using and how many IOPS and stuff. That is very nice if you're using jails at all. Okay, so I guess you all know gstat. It looks like this. It is a top-style
19:31
thing that shows the geom devices and read and write operations per default. It can show deletes and other stuff as well. I actually have no weight. I don't
19:46
because I'm not online. Okay, so this is gstat and these are the numbers I was interested in getting into a graph and that's why we're here today. There's a dash c flag for gstat that
20:07
makes it output in csv mode, it's called, in the man page instead of the top-style display. Unfortunately, a bit of a complaint would be that when you enable dash c, it also enables
20:27
endless mode so it keeps running. You can't just make it output values once and exit. So that should probably be added because usually in exporters, when Prometheus connects, you
20:41
scrape the data and then you return it. But I've had to keep it running endlessly, the gstat and just streaming the output from it into the collector. Also, if gstat could somehow export the counters, it bases the top display on
21:01
so these are the difference between counters and gauges. And we lose a bit of precision because I'm reading the gstat values as counters. It shows me how much there's an update frequency in gstat, so every five seconds or every one second or something. And stuff that happens in between, let's say every five seconds, is kind of lost.
21:22
And if you had the counters instead, Prometheus would be able to, it doesn't matter if you get all the information. And I'm not a C person, but looking at the code, it looks like it is counters when it comes out of the kernel and somewhere it is converted to the human-friendly
21:42
display that we see in gstat. Okay, so with gstat, we really have everything we need. With the dash c mode, we have everything we need to write an exporter. Like I said, the previous client library is easy to get started with. The gstat exporter is 240 lines.
22:08
It sounds like more than it is because it's formatted with these code beautifying tools, and they are very generous with the new lines. So it's on GitHub, and yeah, port 9248
22:22
is the gstat exporter. They don't add them to services files everywhere because they're so dynamic and there's so many of them, but they do have the page on their website or the wiki on GitHub to keep track of which ports are used by what.
22:41
Okay, so running the exporter locally and grabbing for da0 and grabbing for write, this is the output for the gstat exporter. As you see, it's just plain text, and this is fetch outputting to stdout. So it's a plain text response, and it's just
23:02
lines of a metric name and then labels and then finally a space and then a number, which is the actual value for that metric with that combination of labels for that specific time. As you can see, I've stuffed all the info I can get from geom,
23:25
what's it called, geom identifier or something. There's a geom command that can return RPM and all these neat metrics. So I can get, for example, sector size and RPM and the serial number for the disks and the size, which is very nice because as we will see later in
23:44
Grafana, it makes it easy to filter, for example, and show only the NVMe drives or only the spinning drives or only the 12 terabyte drives or whatever you need.
24:00
Okay, so as I said, it doesn't visualize stuff by itself, but the integration to Grafana is really good. Once you add the Prometheus instance as a data source in Grafana, when you start typing a metric name, it autocompletes it and it really works very, very well.
24:21
And you can filter a dashboard by label values and I'll show you what I mean by that later, but that, for example, means that I can easily in Grafana just select one disk if I'm suspecting that one disk is foggy or select only the 5400 RPM drives across all servers if I should for some reason have the need for that. I published the dashboard at the Grafana
24:46
dashboard publishing place and if you don't know Grafana, I highly recommend it. It is very easy to make. It makes pretty graphs for you, like you don't have to decide on
25:00
colors and stuff. It uses colors that look well together and generally just works extremely well. Okay, I'm going to need to fix my internet connection, so talk among yourself for 30 seconds while I get this... Just work for the love of... Sorry, yes, I would if I could.
26:52
It's a long and boring explanation why it doesn't work, but... And usually the phone just works, but of course right now it doesn't.
27:10
Yeah, yeah, thanks guys, but it's not going to work.
27:43
This is great advertisement for Linux. It is a bit, but it really isn't.
28:00
I'm glad I warned that this might happen ahead of time. Yes, okay, so maybe we skip the live
28:25
demo since it's not working. I'm sorry about that. I'll talk a bit more and then try again in a few minutes when it's had time to recover. I do have some screenshots in the beginning that I can go back to, but... Oh, yeah, thanks.
28:46
It was a pretty quick project. I did it in a day, started in the morning investigating how I would do it, and then I wrote the code during the day, and in the evening I did the dashboard, and then I wrote a blog post and posted it, so it was... In the scope of a day,
29:03
you can write an exporter for something, including doing a nice dashboard and making everything work. If GSTAT gets support for outputting just a line and then exiting, I'll probably change the way it calls GSTAT so it doesn't keep it running perpetually, but just calls it when it is
29:22
scraped. Maybe add flags to support if there's a need for it. GSTAT can show consumers as well as producers in the geostack, and some people might need this for reasons I can't really.
29:45
The Grafana dashboard needs some fine-tuning, and if anybody feels like helping with that, then you're very welcome. I'm gonna get this one quick final attempt. Well, I have it open,
30:25
fortunately, so we can show you some of the things I was talking about. The labels I added to the metrics can be used in Grafana to make this nice little table in the top that's
30:44
shown. In the top, we have the filters that Grafana makes it possible to filter by label values, so this shows server names, and this shows the different geomes and the different disk sizes
31:05
we have, and sector sizes, and RPM and stuff. This is where, for example, I didn't know we still had a 5400 RPM drive, but after I installed this, then I got a bit better overview of the entire, and it's a nice way to see, for example, serial number. If you know a disk is
31:24
performing poorly, you can easily see the serial number and ask your on-site hands to fix it. Other than that, there's sections for each of the metrics geome exports, so there's a latency graph for read, write, and delete latency, and there's a bandwidth section for, of course,
31:47
it can't. Trust me, there's a bandwidth graph behind if I was online, and there's an IOPS section, and the queue depth. This is showing 30 days. I've had this running for a few months
32:04
since I wrote it, and one of the fun things is that if you've ran a very large set of S-pools, you probably know that the scrubbing can take almost forever to complete, which means that many people with large pools end up doing a nightly scrub job that just
32:25
pauses it, runs it through the night, and pauses it in the day, in the working hours, and that's a fun. This is the PC percent, and the scrubs are these. This is 30 days, so you can see we scrub all weekend, and then every night, and then all weekend, and every night, so it's a fun pattern to observe, and we found a bad disk in one of the servers using
32:48
this. I can't find the data now, of course, but it works really well to get an overview of the disk utilization, and that's actually all I wanted, so that's very nice.
33:09
Any questions? We have plenty of time. Yes? My OCD screams at me that the media size should be just a number without the string. Yeah, but it's exported directly from the string that geominfo exports, so I could cut
33:27
it off, and then tell Grafana that it's a byte value, and it would humanize it for me. You're right. That would be easy. Grafana knows units, so this graph,
33:43
I've told it that this is a number of seconds, and then it auto-scales the y-axis to show that this is milliseconds, of course, not seconds, fortunately, and it understands megabytes per second and stuff like that, so it is very lovely to work with. Yes?
34:07
Have you had any trouble with Chromethius or Grafana because you're on FreeBSD instead of Linux, like things looking for system B or stuff like that? Apart from it not exporting disk utilization? No. Oh, right. The question was, have I had any problems running a Grafana or
34:24
Prometheus on FreeBSD because it's FreeBSD and not Linux? And no, apart from the node exporter, that's the thing that exposes system metrics like memory and disk IO and stuff like that, apart from it not exporting disk IO and not exporting a set of S data, then no, everything works very well.
34:46
The ports work perfectly and they are updated timely and run very well. The latest version, the upcoming version of node exporter does have FreeBSD set of S support,
35:02
and I'm very much looking forward to that, and combined with this, then you can get a great impression of how your storage is doing. Yeah? Yes?
35:21
The first one is, you were touching keywords like alerting and federation and this kind of stuff, so my question is about these servers what you were scraping, they were only scraping endpoints or they were scraped and processed at the same site,
35:43
or you had one central Chromethius server and then scraping all the endpoints? Yes, one central server that connects out to, you configure it, tell it which targets to scrape, and then it just sucks up all the metrics that are available at that endpoint.
36:09
No, I mean, you would think, but no, it just really works very well. It's a bit of a radical, both the pull model and the idea that it
36:23
does the scrape as it connects, so if Chromethius doesn't connect for a while, then there's no data collected, but it really works very well, and it turns out that it doesn't matter too much as long as you use counters, it would matter for GSTAT exporter, because you would lose the precision if you reboot your Chromethius server, then you would lose the five minutes or whatever
36:43
it takes, but if you have counters available, then you don't, because it doesn't matter if Chromethius is gone for a while, it'll figure out how to, what's the word I'm looking for, put, yeah exactly, it'll smooth it out,
37:03
so no, it works very well, and people are doing some, like the big hosting operators, have some, like they have 500 Chromethius servers doing millions and millions of data points per second, some crazy big instances, so for my users with 100 servers or something,
37:22
I know it's always nice to know that it can handle way more than you need,
37:45
well the alert manager is a separate jail, I mean it could be a separate server if you want it,
38:05
yes, it is, yeah, it is stuff like this, it is infosec, it is also hardware, open source, and it really brought, if you've been to the German CCC camp, or the Dutch
38:23
big hacker camps, it is very much inspired by those, so it is kind of reminds a little bit of first time, except that people live in tents, and there's not as many people of course, about 400 people I think this year, and it's slowly growing, it's the fifth year this time,
38:41
and it's an annual event, and it's getting traction, it's a lovely little event, so any more questions, yes, absolutely, I've used,
39:12
no, not really, I haven't used it enough to, I've not set it up, I've only briefly been exposed to it when somebody else had configured it, so I don't know enough, but
39:25
you know how you can sometimes sense that something has momentum, like a project, like every, I think everywhere I look, Prometheus exporters are popping up like natively inside, whatever, Nginx or whatever, suddenly has a way to explore, and that tells me that,
39:42
you know, I'm not the only one who feels that this is the right approach, and that's very nice, because I think we have been, I've been looking for something nice, for years and years, I've never been really happy, even back when Nagios was the thing to run,
40:00
I've never really been very happy with it, I think everybody really kind of loved to hate it, so I am so happy that finally something, I mean the aesthetics are also very important, because even though technically it's the same numbers, if it's an ugly graph or pretty graph, but it just, it still makes a difference, it's nicer to work with.
40:23
One response to that, so just looking at influx, while it's nice to work with in at least some aspects, Prometheus has much much more advanced math available, like for example in influx, it's not possible in a single query to get, for example you get the return codes of HTTP
40:47
requests, in influx it's not possible to get what proportion of requests were say errors, it's possible in graphite, it's possible in Prometheus, it's not possible in influx.
41:10
Sure, any more questions? I have a quick question, because I'm rarely surrounded by a lot of people, so I just, oh damn, I went offline, oh and I can't even show it, okay, well the
41:34
question though, the problem is, you know, the jail exporter I was talking about, which
41:41
uses RCTL to export jail metrics, and then you can do graphs for example memory use inside a jail, so I use that of course, because I use jails for all my stuff, and it's very nice to if a disk is being trashed, you can easily see which jail is causing this,
42:01
Postgres doing an auto vacuum, or what's going on, RCTL counts shared memory twice in jails, or as many times, I have, what I have in this tab that I can switch over to, because I'm offline, I have two jails on this server, it has 128 gigabytes of ram, I have two Postgres jails, and RCTL
42:26
tells me that one jail is using about 130 gigs, and the other jail is using 80 gigs, which is about 200 gigs, and there's 40 more jails on it, so clearly something is wrong, and I've been able to track down as much as it is, I don't know
42:42
what the fix is, but it's absolutely misreporting memory use in jails, and since if I have 100 Postgres workers, and each of them is using two gigabytes of ram, then, and most of it is shared, what you want RCTL to export is how much memory would be freed if I shut the jail down, I don't care if 100 workers is using two gigabytes, you shouldn't report 200 gigabytes
43:04
memory use, does anybody know, is this just a bug, or is there, is this not possible, what I'm asking for, or, I mean. If it's memory shared between jails, are you even able to tell? Not between jails, between 100 Postgres workers in the same jail,
43:20
for example, and then each of those, they have access to the same shared memory space, and it's been counted 100 times because there's 100 workers, so it tells me that one jail is using 140 gigs of ram on a 128 gig server, which clearly is not true,
43:47
it would, but we can talk about it later after, I just, it's been bugging me because I finally got some nice jail graphs and the memory usage is just way, way off, I was like,
44:03
what the hell is wrong, how can it be using 300 gigabytes of ram when it has 128, and RCTL exports the V memory use as well, and that's, of course, a high number and higher than the, it's 1.2 terabytes of memory, V memory, I think, but the memory use should be, we agree that
44:21
it should be however much would be freed if the jail was shut down, right, otherwise what's the point of reporting them, I mean, you see my, you see what I mean, okay, I think I'll open up PR and write to a mailing list and see if I can, I think I can catch somebody's attention
44:41
with that, but other than that, the jail exporter is really awesome and stuff like IOPS and that works very well, so you can see which jail is trashing a disk or using a lot of CPU, I think that's it for me, we're done with a quarter of an hour to spare, so you can use that time
45:05
for self-reflection and self-improvement, and thanks for having me, and I hope you enjoyed first time and my talk, and yeah, thank you.