OpenTelemetry with Grafana
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62001 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023107 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
LoginBit rateOpen sourceError messageCartesian coordinate systemSpring (hydrology)Open setMultiplication signMetric systemType theoryBitDatabaseService (economics)Demo (music)Order (biology)Computer architectureBootingDependent and independent variablesStandard deviationJava appletData managementWordSlide ruleSource codeVirtual machineLaptopProgram flowchart
03:25
MIDIMaxima and minimaColor managementRadio-frequency identificationThermal expansionMoment of inertiaTelephone number mappingFunctional (mathematics)Bit rateMetric systemLine (geometry)Query languageCodeHistogramMultiplication signSpeicherbereinigungNumberPhysical systemStructural loadSummierbarkeitCartesian coordinate systemServer (computing)Open setCountingDistribution (mathematics)Drop (liquid)BitType theoryQuantum stateTheoryOpen sourceService (economics)Online helpRaw image formatWindowTheory of relativityDemo (music)Asynchronous Transfer ModeDiagram
08:36
Execution unit2 (number)Visualization (computer graphics)Parameter (computer programming)Different (Kate Ryan album)Service (economics)HistogramSystem callQuery languageProgramming languageMultiplication signAttribute grammarServer (computing)Cache (computing)Error messageShape (magazine)Open setSingle-precision floating-point formatCartesian coordinate systemStandard deviationMereologyMetric systemTracing (software)Drop (liquid)Drag (physics)Line (geometry)Open sourceGoodness of fitBitRepresentation (politics)Thresholding (image processing)Physical systemPrice indexStructural loadDistribution (mathematics)Semantics (computer science)WindowSoftware developerComputer animation
13:47
Computer virusView (database)Computer fileExtension (kinesiology)Slide ruleClient (computing)GoogolFormal grammarServer (computing)Tracing (software)Tracing (software)Client (computing)Game controllerService (economics)Dependent and independent variablesCorrespondence (mathematics)Server (computing)Software developerCartesian coordinate systemOpen setSpring (hydrology)Point (geometry)BootingSingle-precision floating-point format2 (number)Different (Kate Ryan album)Multiplication signJava appletLine (geometry)Computer animation
15:57
Graph (mathematics)Gamma functionDew pointMaxima and minimaComputer virusMKS system of unitsJava appletSpring (hydrology)Service (economics)CausalityLine (geometry)RootGame controllerMetadataCodeAttribute grammarError messageEvent horizonWindowStandard deviationSpring (hydrology)Type theoryTheory of relativityComputer animation
17:41
IntelExecution unitLocal area networkComputer virusGeometryAdvanced Boolean Expression LanguageColor managementType theoryRight angleService (economics)CASE <Informatik>WindowDependent and independent variablesLevel (video gaming)Order (biology)Graph (mathematics)MathematicsTracing (software)Probability density functionCartesian coordinate systemGoodness of fitMultiplication signOpen setMetric systemLoginDiagramCausalityRootDot productComputer architectureSystem call1 (number)BitOpen sourceDatabaseQuery languageStatisticsFront and back endsException handlingPhysical systemInformationSoftwareError messageIntrusion detection systemDiagram
21:45
Gamma functionElectronic data interchangeQuantumGEDCOMLine (geometry)Maxima and minimaMenu (computing)Exception handlingError messageMultiplication signSpring (hydrology)Bit rateMetric system2 (number)LoginImage resolutionCASE <Informatik>Different (Kate Ryan album)Functional (mathematics)InformationElectronic mailing listTracing (software)Query languageDemo (music)Computer animation
23:30
Equals signMaxima and minimaExecution unitComputer iconBuildingRow (database)RepetitionDynamic random-access memoryComputer virusGamma functionContext awarenessFile formatService (economics)CASE <Informatik>InformationClosed setLoginBitOpen setMetric systemLine (geometry)Cartesian coordinate systemKey (cryptography)Correspondence (mathematics)Java appletPrice indexSource code
25:21
Computer fileView (database)Extension (kinesiology)Slide ruleFormal grammarCartesian coordinate systemMetric systemOpen setAttribute grammarBuffer solutionService (economics)Projective planeTracing (software)Group actionProcess (computing)LoginSoftwareWeb browserComputer architectureStandard deviationFront and back endsVirtual machineInstance (computer science)Type theoryDirection (geometry)Parameter (computer programming)MereologyMiniDiscNetwork topologyWeb pageBitProduct (business)Java appletArea
31:35
Program flowchart
Transcript: English(auto-generated)
00:06
Okay, hello everyone. Welcome to the talk on open telemetry with Grafana. The microphone broke so I need to do it with this microphone now. Let's see how it goes with typing and live demo.
00:21
A few words about me. So who am I? Why am I here talking about Grafana and open telemetry? So I work at Grafana Labs. I'm an engineering manager and I'm also manager for our open telemetry squad. And I'm also active in open source so I'm a member of the Prometheus team where I maintain the Java metrics library.
00:43
So what are we going to do in this talk in the next 25 minutes or so? So it will almost exclusively be a live demo. So basically the idea is I have a little example application running on my laptop and it is instrumented with open telemetry. I will show you in a minute what it does and how I instrumented it.
01:01
And I also have an open source monitoring backend running. It consists of three databases. One is Loki which is an open source logs database. One is Tempo which is an open source trace database. And one is Mimir which is an open source metrics database. So Mimir is compatible with Prometheus so it could have shown the exact
01:24
same demo using Prometheus instead of Mimir so it doesn't really matter for now. And of course I also have Grafana. I have those databases configured as data sources. And what we are going to do, we are going to start up Grafana, you know, have a look at metrics, have a look at traces, have a look at logs.
01:42
And basically the idea is that at the end of the talk you kind of have seen all the signals that come out of open telemetry. You know, explored a bit what you can do with this type of data and so you should have a good overview how open source monitoring with open telemetry looks like. Right? So last slide before we jump into the live demo.
02:03
So this is just a quick overview of what the example application does so that you know what we are going to look at. It's a simple hello world REST service written in Java using Spring Boot. And so basically you can send a request to port 8080 and it will respond with hello world.
02:22
And in order to make it a bit more interesting I made it a distributed hello world service. So it doesn't respond directly but when it receives a request it reaches out to a greeting service running on port 8081. The greeting service responds with the greeting which is hello world and then the response is forwarded to the client.
02:40
Right? And there are random errors to have some error rates as well. So basically a hello world microservice architecture or whatever. And in order to instrument this with open telemetry I used the Java instrumentation agent that's provided by the open telemetry community. That's something you can download on GitHub.
03:01
And the thing is this thing you basically attach it to the Java virtual machine at startup time with a special command line parameter. So I didn't modify any source code. I didn't use any SDK or introduce any custom stuff. All we are going to look at in this demo is just data produced by attaching the open telemetry instrumentation to a standard Spring Boot application.
03:24
Right? Cool. So let's get started. As said I have my data sources configured here. So Prometheus and Mimir are compatible so it doesn't really matter which one we choose. There are a lot of, so I want to start with metrics.
03:42
And yeah, is there, yeah. Can we turn the lights down a bit? I don't know. Okay.
04:03
Maybe the other ray ran. Okay. I will just continue. Come on. So there are lots of metrics that you get from the open telemetry instrumentation. So kind of JVM related stuff like garbage collection activity and so forth.
04:25
But the one I want to look at. Oh no. It's getting brighter and brighter. Yay. Yay.
04:43
Okay, great. I think there is also a light mode in Grafana. Maybe that would have been a better choice. But no, I'm not going to use light mode. So let's figure out how to do the demo while I have a microphone that I should hold in my hands.
05:02
Let's just put it here. Okay, thank you. Cool. So the metric we are going to look at for the demo, it's a metric named HTTP server duration. This is a metric of type histogram.
05:23
So histograms have a couple of different numbers attached to them. So there are histogram buckets with the distribution data and so forth. And there's also a count. The count is the most simple one. So we are going to use this in our example. I actually got it two times. I got it once for my greeting service here and once for the Hello World application.
05:44
And if we are just, you know, running this theory. Maybe take a little bit of a shorter time window here. Then we basically see two request counters, right? One is the green line which is counting the request residing in HTTP such as 200.
06:01
So the successful requests. And basically we see that since I started the application on my laptop, I got about a little more than 400 successful requests. And the yellow line is, you know, request residing in HTTP such as 500. And we got up around 50 of them, right? And obviously raw counter values are not very useful, right?
06:23
Nobody is interested in how often was my service called since I started the application. And the way, you know, metric monitoring works with Prometheus, as probably most of you know, is that you use the Prometheus query language to get some useful information out of that kind of data, right?
06:41
And I guess most of you have run some Prometheus queries but are still going to show maybe a couple of examples. So for those of you who are not very familiar with that, does this one work again? Hey, nice. It's even better. The lights work, the microphone works. Wow. Now let's hope the demo works.
07:03
So, I'm going to run just a couple of quick, you know, Prometheus queries so that for those of you who are not very familiar with it, so that you get an idea of what it is, right? And the most important function in the Prometheus query language is called the rate function. And what the rate function does, it takes a counter like this and a time interval like five minutes
07:24
and then it calculates a per second rate, right? So based on a five minute time interval, we now see that we have about 0.6 requests per second residing in HDP status 200 and we have about 0.1 requests per second residing in HDP status 500.
07:42
And this is already quite some useful information, right? So typically you want to know the total load on your system, not by status code or something. So you basically want to sum these two values up and obviously there's also a sum function to sum values up. And if you call that, you get the total load on your system which is just one line now
08:03
and it's just, you know, around 0.7 requests per second, right? And this is, yeah, this is basically how Prometheus queries work. If you're not familiar with the syntax, there's also kind of a graphical query builder where you can, you know, use a bit drag and drop and get a bit more help and so forth, right?
08:21
And so eventually, you know, when you got your queries and get your metrics, so what you want to do is you create a metrics dashboard. And for monitoring HDP services, there is, there are a couple of best practices, what type of data you want to visualize on a dashboard for monitoring HDP services. And the most simple and straightforward thing is to visualize three things.
08:44
One is the request rate, so for the current load on the system which is exactly the query that we are seeing here. The next thing you want to see is the error rate, so the percentage of calls that failed. And the third thing is duration, how long does it take, right?
09:01
And I created a simple example dashboard just to show you how this looks like. So I put the name of the service as a parameter up here so we can reuse the same dashboard for both services. Maybe let's use a 15-minute time window, so here I started the application. The first is the request rate, that's the exact same query that we just saw.
09:22
Second thing here is the error rate, so we have about, I don't know, around 10% errors in my example application. And then for duration, there are a couple of different ways how to visualize that. So what we see here is basically the raw histogram, right, the histogram buckets. And this representation is actually quite useful because it shows you the shape of the distribution.
09:46
So what we see here is two spikes, one around 600 milliseconds and one around 1.8 seconds. And this is a typical shape that you would see if your application uses a cache, right? Because then you have a couple of requests that are responded quite quickly, those are the cache hits.
10:05
Couple of requests are slow, that are the cache misses. And visualizing the shape of the histogram helps you understand kind of the latency behavior of your application, right? The other and most popular way to visualize curations is this one here.
10:21
These are percentiles, so the green line is the 95th percentile, so it tells us 95% of the calls have been faster than 1.7 seconds and 5% slower than that. The yellow line is the 50th, so half of the calls faster than that, half of the calls slower than that. And this doesn't really tell you the shape of the distribution, but it shows you a development over time, which is useful as well.
10:45
So if your service becomes slower, those lines will go up, right? And it's also a good indicator if you want to do, you know, alerting and so forth. You can define a threshold and say it's above, if it's above a certain threshold, I want to be notified and stuff like that. And there are other more, you know, experimental things like this heat map,
11:04
showing basically development of histograms over time and stuff like that. So it's pretty cool to play with all the different visualizations in Grafana and, you know, see what you can get. So this is a, you know, quick example of a so-called red dashboard, request rates, error rates, duration based on open telemetry data.
11:25
And the cool thing about it is that it actually, all that we are seeing here is just based on that single histogram metric HTTP server duration. And the fact that this metric is there is not a coincidence.
11:42
The metric HTTP server duration is actually defined in the open telemetry standard as part of the semantic conventions for HTTP services. So whenever you monitor an HTTP server with open telemetry, then you will find a histogram named HTTP server duration.
12:00
It will have the HTTP status as an attribute. It will contain the latencies in milliseconds. That's all part of the standard. So it doesn't matter what programming language your services uses, what framework, whatever. If it's being monitored with open telemetry and it's compatible, you will find that metric and you can create a similar dashboard like that.
12:23
And this is kind of one of the things that make application monitoring with open telemetry a lot easier than it used to be before these standardization. Good. So that was a quick look at metrics. But of course, we want to look at the other signals as well.
12:41
So let's switch data sources for now and have a look at traces. So tracing again, there's a kind of such like graphical search where you can, you know, create your search criteria with drag and drop. There's a relatively new feature, which is a query language for traces. So going to use that for now.
13:02
And, you know, one thing you can do is to just, you know, search by labels. So I can, for example, say I'm interested in the service name greeting service. And then I could basically just open a random trace here. Let's take this as an example.
13:21
Can I need to zoom out a little bit to be able to close the search window here? OK. So this is how a distributed trace looks like. And if you see it for the first time, it might be a bit hard to understand, but it's actually fairly easy. So you just need like two minutes of introduction and then you will understand traces forever.
13:44
And to give you that introduction, I actually have one more slide. So just to help you understand what we are seeing here. And the thing is distributed traces consist of spans, right? And spans are time spans. So a span is something that has a point in time where it starts and a point in time where it ends.
14:04
And in open telemetry, there are three different kinds of spans. One are server spans. The second is internal spans. And the third is client spans. OK. So what happens when my Hello World application receives a request? So the first thing that happens if a server receives a request, the server span is created.
14:24
So that's the first line here. It's started as soon as the request is received. It remains open until the request is responded, right? Then, and as I said in the introduction that I used Spring Boot for implementing the example application.
14:40
And the way Spring Boot works is that it takes the request and passes it to the corresponding spring controller that would handle the request. And open telemetry's Java instrumentation agent is nice for Java developers because it just creates internal spans for each spring controller that is involved. Right? And that is the second line that we are seeing here.
15:01
It's basically opened as soon as the spring controller takes over and remains open until the spring controller is done handling the request. Which might seem not too useful if I have just a single spring controller anyway. But if you have kind of a larger, you know, application and you have multiple controllers involved, it gives you quite some interesting insights into what's happening inside your application.
15:24
Like you would see immediately like which controller do I spend most time in and so forth. Right? And then eventually my Hello World application reaches out to the greeting service and outgoing requests are represented by client spans. So the client span is basically opened as soon as my HTTP request goes out and remains open until the response is received.
15:46
And then in the greeting service the same thing starts again. You know, request is received which creates a server span and then I have a spring controller as well which is an internal span and that's the end of my distributed application here. And this is exactly what we are seeing here.
16:01
And each of those span types has a corresponding metadata attached to it. So if you look at one of the internal spans here we see the name of the spring controller and the name of the controller method and a couple of JVM related attributes, whatever. And if you look at an HTTP span for example we see of course HTTP attributes like the status code, method and so forth.
16:25
So, of course you do not want to just look at random spans. So usually you are looking for something. There are standard attributes in OpenTelemetry that you can use for searching. So we already had the service name greeting service for example.
16:43
The most important or one of the most important attributes is HTTP status code. This one here. And if we for example search for spans with HTTP status code 500 then we should find an example of a request that failed.
17:01
So let's close the search window again. Yes, that's an example of a failed request. You see it with the indicated by those red exclamation marks at the bottom here. So this is where the thing failed, right? So the root cause of the error is the internal span, something in my spring controller in the greeting service.
17:21
If I look at the metadata attached to that I actually see that the instrumentation attached the event that caused the error. And this even includes the stack trace. So you can basically immediately navigate to the exact line of code that is the root cause of this error, right? And this is quite cool.
17:41
So if you have a distributed application and you get an unexpected response from your hello world application, without distributed tracing it's pretty hard to find that actually there is an exception in the greeting service that propagated through your distributed landscape and then eventually caused the unexpected response.
18:02
And with distributed tracing finding these kind of things becomes pretty easy because you get all the related calls grouped together, you get the failed ones marked with an exclamation mark and you can pretty easily navigate to what's the root cause of your error, okay? Cool.
18:20
So that was a quick look at traces. There are a lot of interesting things about tracing. Maybe one thing I would like to show you because I find it particularly cool. So if you have all your services instrumented with tracing in your backend, then basically those traces give you metadata about all the network calls happening in your system.
18:42
And you can do something with that type of data, right? So for example you can calculate something that we call the service graph. So it looks like this. It's maybe not too impressive if you just have two services calling each other, right? So but if you imagine, you know, a more larger, you know, dozens or hundreds of services,
19:01
so it will generate a map of all the services and indicate which service calls which other service. And this is quite useful for example if you intend to deploy a breaking change in your greeting service and you want to know who's using the greeting service, what would I break? Then looking at the service graph you basically get this information right away.
19:22
Traditionally if you don't have that, you basically have a PDF with your architecture diagram and then you look it up there. And also traditionally there's at least one team that deployed something and forgot to update the diagram and then you missed that. And with the service graph that won't happen, right? This is the actual truth. This is based on what's actually happening in your backend
19:42
and this is pretty useful in these situations, right? And you can do other things as well like, you know, have some statistics like the most frequently called endpoint or the endpoint with the most errors and stuff like that. So that was a quick look at traces.
20:01
So we covered metrics, we covered traces. One thing I want to show you is that metrics and traces are actually related to each other, right? And so in order to show that I'm going to go back to our dashboard because if you, let's take a 15 minute window then we get a bit more examples. So if you look at the latency data here, you notice these little green dots.
20:24
These are called exemplars and this is something that's provided by the auto instrumentation of open telemetry. So whenever it generates latency data, it basically attaches trace IDs of example traces to the latency data.
20:40
And this is visualized by these little green dots, right? And so you see some examples of particularly fast calls, some examples of particularly slow calls and so forth. And if you for example take this dot up here which is kind of slower than anything else, it's almost two seconds, right? Then you have the trace ID here and you can navigate to tempo and have a look at the trace
21:02
and start figuring out why did I have an example of such a slow call in my system, right? And in that case you would immediately see that most of the time spent in the greeting service. So if you're looking for the performance bottleneck then this is the most likely thing. Yeah, four minutes? That's fine.
21:21
Good. So if I have four minutes, it's high time to jump to logs, the third signal that we didn't look at yet. So let's select Loki, our open source logs database as a data source. So again, there's a query language, there's a graphical query builder and so forth.
21:41
So let's just open random logs coming from the greeting service. Looks a bit like this. So even, I didn't even log anything explicitly. I just turned on some whatever spring request logging so that I get some log data. And from time to time I throw an exception which is an IO exception to simulate these errors.
22:01
Looks a bit broken but that's just because of the resolution that I have here. Yeah, so what you can do, of course you can do some full text search for example. You can say I'm interested in these IO exception and then you would basically get,
22:22
if you spell it correctly, like that, then you would get the list of all IO exceptions which in my case are just the random errors I'm throwing here. And this query language is actually quite powerful. So you can, this is kind of filtering by a label and filtering by full text search
22:40
but you can do totally different things as well. For example, you can have queries that derive metrics based on log data. There's a function pretty similar to what we have seen in the metrics demo which is called the rate function. So the rate function again takes a time interval and then calculates the per second increase rate.
23:01
So it basically tells you that we have almost 0.1 of these IO exceptions per second in our log data which is also kind of useful for information to have, right? And the last thing to show you because that's particularly interesting is that these logs and traces and metrics are again not independent of each other.
23:26
They are related to each other and so if you look at an example here, just let's open a random log line. So what we see here there's a trace ID and this is interesting. So how does a trace ID end up in my log line?
23:40
So this is actually, so also a feature of the Java instrumentation that's provided by the open telemetry community. So the way logging in general works in Java is that there's a global thing with key value pairs called the log context and applications can put arbitrary key value pairs into that context
24:00
and when you configure your log format you can define which of those values you want to include in your log data. And if you have this open telemetry agent attached then as soon as a log line is written in the context of serving an HTTP request then the corresponding trace ID is put into that log context and you can configure your log format to include the trace ID in your log data.
24:23
And that's what I did and so each of my log lines actually has a trace ID and so if I see something fancy and I want to know maybe somewhere down my distributed stack something went wrong I can just query that in tempo, navigate to the corresponding trace, close that here, yeah?
24:41
And then basically maybe get some information what happened. And then the same navigation works the other way around as well. So of course there's a little, you know, log button here so if I see something fancy going on in my greeting service thing here and maybe the logs have more information I can click on that, navigate to the logs and then it basically just generates a query, right?
25:02
I click on the greeting service with that trace ID so it's basically just a full text search for that trace ID and so I will find all my corresponding log lines. In that case just one line but if you have a bit better logging then maybe it would give you some indication what happened there. Okay, so that was a very quick 25 minutes overview of, you know, looking a bit into metrics
25:28
looking a bit into tracing, looking a bit into logs. I hope it gave you some impression, you know, what's the type of data that you get out of open telemetry looks like. All of what we did is really, you know, without even modifying the application.
25:43
I didn't, you know, even start with custom metrics, custom traces and so forth so but it's already quite some useful data that we get out of that. If you like the demo, if you want to explore it a bit more, want to try it at home I pushed it on my GitHub and there's a readme telling you how to run it
26:01
so you can do that. And yeah, next up we have a talk that goes a bit more in detail into the tracing part of this and then after that we have a talk that goes a bit more into detail how to run open telemetry in Kubernetes, so stay here and thanks for listening.
26:25
Please remain seated during Q&A, otherwise we can't do a real Q&A, so please remain seated. Are there any questions? Yes.
26:43
Hi, thank you for this. One quick question. You mentioned you just need to add some parameters to the Java virtual machine to run the telemetry. What happens to my application if, for example, the back end of the telemetry is down? Is my application failing or impacted in any way?
27:03
If the monitoring back end is down. Yes, say the monitoring is down but I started my application with these parameters. Is it impacting the application? No, I mean you won't see metrics of course if your monitoring back end is down but the application would just continue running. So typically in like production setups the applications wouldn't send telemetry data directly to the monitoring back end
27:29
but what you usually have is something in the middle. There's alternatives, there's the Grafana agent that you can use for that, there's the open telemetry collector that you can use for that and it's basically a thing that runs close to the application,
27:43
takes the telemetry data off the application very quickly and then, you know, can buffer stuff and process stuff and send it over to the monitoring back end and that's used for decoupling that a little bit, right? And if you have such an architecture the application shouldn't be affected at all by that.
28:02
Two more questions. Yeah. Two more. So I really like the being able to link from your metrics to traces but what I'm actually really curious to be able to do and as far as I know doesn't exist or I guess that's my question is like
28:21
is there any thought towards doing this is being able to go the other direction where what I'd like to be able to answer is here's all my trace data and this node of the trace incremented these counters by this much so I could ask things like how much network IO or disk IOPS did this complete request do
28:44
and where in the tree did that occur? Yeah, that's a good question. I mean linking from traces to metrics is not so straightforward because I think the things you can do to relate this is to use the service name
29:03
so if you have the service name part of your resource attributes of the metrics and consistently you have the same service name in your trace data then you can at least you know navigate to all traces coming to all metrics coming from the same service. Maybe you have some more you know related attributes like in whatever instance ID and so forth
29:24
but it's not like really a one-to-one relationship. So yeah, no, I don't think that's possible. So in this example you've shown that Grafana works great with server-side applications.
29:44
Have you had examples of client-side applications, mobile, embedded desktop applications that use Permafield metrics and then ship their metrics and traces to the metric backend?
30:02
Did I hear it correctly? You're asking about starting your traces on the client-side in the web browser and stuff? So if you have tracing on the server-side, what about having traces and metrics on the client-side for example for an embedded or mobile application so that you could actually see the trace
30:21
from when the customer clicked a thing and see the full customer journey? That's a great question. That's actually an area where there's currently a lot of research and new projects and so forth. So there is a group called Real User Monitoring, RUM, in OpenTelemetry that deal with client-side applications.
30:42
There's also a project by Grafana. It's called Faro. It's kind of JavaScript that you can include in your front-end, in your HTML page and then it gives you traces and metrics coming from the web browser and this is currently a pretty active area.
31:02
So lots of movement there. So there are things to explore. So if you like, check out Faro. It's a nice new project and standardization is also currently being discussed but it's newer than the rest of what I showed you. So there's no clear standard yet or nothing decided yet.
31:26
Cool. Okay. Thanks everyone again.