HawkTracer profiler
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47400 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020204 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
VideoconferencingPrime idealMetropolitan area networkProfil (magazine)Prime idealVideoconferencingMobile appMultiplication signFocus (optics)Computer animation
00:41
Integrated development environmentPrime idealVideoconferencingComputing platformSpacetimeMeasurementOverhead (computing)Installation artPlastikkarteConsistencyElectronic mailing listProfil (magazine)Kernel (computing)SpacetimeKeyboard shortcutPrototypePresentation of a groupProjective planeCartesian coordinate systemMultiplication signPrime idealDifferent (Kate Ryan album)PlastikkarteBefehlsprozessorSoftware developerIntegrated development environmentGame theoryComputing platformFormal languageTerm (mathematics)Mobile appBasis <Mathematik>Module (mathematics)Process (computing)Internet service provider2 (number)VideoconferencingMeasurementSemiconductor memoryTracing (software)Metric systemConsistencyTopological vector spaceFinite differenceGroup actionComputer configurationDemonLevel (video gaming)Database normalizationPlotterPhysical systemComputing platformSocial classCellular automatonSystem callDot productCompilerFunctional (mathematics)Workstation <Musikinstrument>Utility softwareMoment (mathematics)Computer animation
04:57
ArchitectureGraphical user interfaceEvent horizonExecution unitType theoryStreaming mediaClient (computing)Binary fileRepresentation (politics)Core dumpCommunications protocolForm (programming)Cartesian coordinate systemEvent horizonGraphical user interfaceVisualization (computer graphics)Client (computing)Internet service providerSoftware developerLibrary (computing)Streaming mediaProfil (magazine)Graph (mathematics)Demo (music)BitDiagramLeakComputer animation
05:36
DiagramEuclidean vectorEvent horizonData bufferThread (computing)StapeldateiData structureStreaming mediaLibrary (computing)Core dumpClient (computing)Software developerMaxima and minimaAmicable numbersDiagramBuffer solutionEvent horizonLibrary (computing)Default (computer science)Mobile appComputer fileStreaming mediaDemo (music)Cartesian coordinate systemKeyboard shortcutClient (computing)Data structureFunctional (mathematics)Serial portData conversionBitSoftware developerFormal languageData storage deviceComputer animation
06:58
Library (computing)Personal digital assistantInstance (computer science)Core dumpStreaming mediaEuclidean vectorDiagramEvent horizonData bufferThread (computing)StapeldateiClient (computing)Data structureSoftware developerType theoryField (computer science)Inheritance (object-oriented programming)Inclusion mapIntegerDecimalMultiplication signBuffer solutionBitProcess (computing)MultiplicationConfiguration spaceInstance (computer science)Type theoryClient (computing)Event horizonMacro (computer science)Functional (mathematics)Field (computer science)Projective planeSocial classExecution unitSet (mathematics)Parameter (computer programming)Category of beingPointer (computer programming)Dot productDesign by contractData structureComputer programmingDifferent (Kate Ryan album)CausalityInsertion loss2 (number)Streaming mediaGroup actionComputer animationSource code
09:19
Core dumpDiagramEuclidean vectorEvent horizonData bufferThread (computing)StapeldateiLibrary (computing)Client (computing)Data structureStreaming mediaSoftware developerLink (knot theory)InformationType theoryConvex hullMaß <Mathematik>Lie groupThomas KuhnInformationEvent horizonClient (computing)Social classFluid staticsIdentifiabilityStreaming mediaType theoryString (computer science)Multiplication signKlassenkörpertheorieAuthorizationNumberMacro (computer science)RandomizationHash functionMetadataFunctional (mathematics)Level (video gaming)Field (computer science)Formal languageSynchronizationParsingThread (computing)Local ringKeyboard shortcutRevision controlData streamEndliche Modelltheorie40 (number)Point (geometry)GodMoment (mathematics)CASE <Informatik>Context awarenessDemo (music)Object (grammar)Identity managementComputer animation
13:45
Resource allocationClient (computing)Demo (music)Data streamRead-only memoryComputer programmingSemiconductor memoryGraph (mathematics)Uniform resource locatorTracing (software)Operator (mathematics)Client (computing)Cartesian coordinate systemVisualization (computer graphics)Event horizonStreaming mediaResource allocationComputer animation
14:06
Convex hullOvalExecution unitInstallation artInstant MessagingMacro (computer science)Event horizonBoolean algebraMIDIContext awarenessIntegerRead-only memoryScheduling (computing)Task (computing)VoltmeterEmpennageBroadcast programmingDuality (mathematics)Mass flow rateVirtual realityCountingAnnulus (mathematics)StrutBuildingPunched cardGamma functionClient (computing)Resource allocationTimestampEvent horizonClient (computing)BitSemiconductor memoryStreaming mediaResource allocationElement (mathematics)Functional (mathematics)Type theoryComputer configurationGraph (mathematics)Virtual memoryIP addressCountingMacro (computer science)Physical systemMultiplication signContext awarenessTupleInformationBuffer solutionComputer programmingLibrary (computing)PlanningField (computer science)Queue (abstract data type)Inheritance (object-oriented programming)Visualization (computer graphics)TimestampUniform resource locatorObject (grammar)MereologyEndliche ModelltheorieAddress spaceDeclarative programmingValue-added networkComputer animation
17:11
Semiconductor memoryResource allocationCountingComputer programmingCartesian coordinate systemLoop (music)Client (computing)Computer animation
17:34
WindowRead-only memoryVirtual realityEvent horizonComputer fontBoolean algebraTask (computing)Scheduling (computing)Context awarenessBroadcast programmingInstant MessagingResource allocationMultiplication signMetropolitan area networkGraph (mathematics)Endliche ModelltheorieEvent horizonComputer animation
17:59
Instant MessagingScheduling (computing)Task (computing)Read-only memoryContext awarenessLibrary (computing)Demo (music)Formal languageComputer-generated imageryThomas KuhnEvent horizonGraph (mathematics)NumberDemo (music)Multiplication signReal numberSystem callCASE <Informatik>Formal languageSemiconductor memoryMedical imagingFront and back endsLibrary (computing)CodeCartesian coordinate systemOperator (mathematics)Interpreter (computing)Particle systemOpen setMereologyClient (computing)Computer animation
19:13
Instance (computer science)Computer-generated imageryCodeThread (computing)Pattern languageRotationString (computer science)Data bufferDefault (computer science)Computer fileBit rateConvex hullFluid staticsAngleCore dumpFunctional (mathematics)Library (computing)Medical imagingRotationComputer fileClient (computing)Interpreter (computing)Multiplication signWrapper (data mining)BitMacro (computer science)Universal product codeIntegrated development environmentExecution unitThread (computing)MereologyPoint (geometry)Tracing (software)Computer animation
21:00
Source codeAttribute grammarLine (geometry)Shared memoryComputer-generated imageryRotationTotal S.A.Local ringComputer fileHill differential equationRankingRotationHoaxAngleComputer programmingComputer fileHookingComputer animation
21:23
Core dumpAttribute grammarComputer fileLine (geometry)Shared memoryAnalog-to-digital converterTracing (software)Binary fileVariable (mathematics)Integrated development environmentCore dumpStreaming mediaData conversionGraphical user interfaceMedical imagingHookingValue-added networkComputer fileComputer animation
22:16
RotationError messageLine (geometry)Source codeFunction (mathematics)Shared memoryTrailBeat (acoustics)RotationComputer fileThread (computing)BitPoint (geometry)Computer programmingMedical imagingComputer animation
23:06
Analog-to-digital converterFunction (mathematics)Attribute grammarLine (geometry)Source codeShared memoryTracing (software)RotationComputer-generated imageryCore dumpGraph (mathematics)Graph (mathematics)Computer programmingCASE <Informatik>Event horizonView (database)Computer animation
23:27
Event horizonKeyboard shortcutGeneric programmingFile viewerWebsiteCodeRepository (publishing)Analog-to-digital converterPlanningEvent horizonCASE <Informatik>Projective planeLink (knot theory)Computer fileAdditionData conversionMoment (mathematics)Keyboard shortcutDemo (music)Type theoryWebsitePower (physics)Multiplication signComputer animation
25:00
FacebookPoint cloudOpen source
Transcript: English(auto-generated)
00:05
Hi, everybody. Thanks for coming. I know for some of you it's quite early yet. My name is Marcin. I work for Prime Video. And this talk will be about the profiler that we did to fix performance issues in Prime
00:21
Video app that's running on living room devices. And I'm not going to talk a lot about the internals of the profiler. I'd rather focus on how to use it, how you can use it, and adapt to your project. But if you want to know more about internals, I'll be here around till the end of the day. So you can just speak to me any time you want.
00:41
So I'll start with explaining why we actually started the new profiler. Because as you might know, there's quite a lot of them on the market already as LTTNG, F-Trace, UF-Trace, S-Trace, Perf, and a lot of others. The thing is that the development environment that we had to work in was quite different than what
01:01
you might get used to. So we've been doing applications that's running on the living room devices, such as game consoles, smart TVs, streaming sticks. And some of the platforms are very friendly for developers. Like game consoles, they are amazing. They provide lots of tools, debugging, utilities, and so on. But smart TVs are not that friendly for developers.
01:23
So very often, all we have is compiler. And we can't really deploy any tool to that device. It's very close. So all we can do is basically deploy a single app. That's our PrimeVDA app. And that's it. So we can't, for example, deploy S-Trace or LTTNG.
01:42
We can't deploy some kernel modules and so on. So it's a very, very limited environment. And also, our application, the PrimeVDA application, is built using three different languages. So we have a native layer, which is a C++. And then that native layer exposes two scripted engines,
02:05
one for Lua and one for JavaScript. And we run them at the same time. So they're basically three different languages. And because some of those platforms are very low in terms of the CPU and memory, we had quite a few performance
02:22
issues, especially on the cheaper platforms. And we wanted to debug it. But as I said, there was not much tools for debugging. There were some tools, but they didn't really meet our requirements. So we said, OK, maybe we can just quickly prototype something, and we'll see if that works. Yeah, as you see, this presentation
02:42
is about the profiler, so it actually worked. So we came up with a list of features that we want to have from our profiler. This is just a short list, but there's quite a few more features. I just listed the more important. So user space and instrumentation base, we basically don't have access to kernel by any means.
03:04
So user space for us was the only option. Instrumentation base, we can't really deploy a second daemon that, I don't know, for example, checks traces every second or so. So everything had to be built into the application. Since our application is written in C and JavaScript
03:20
and Lua, we decided that we're going to write it in C++ or C, but it should be available for those languages as well somehow by providing a bindings layer and so on. Also, because we port to different platforms, those platforms are very different to each other. Some of them have different operating systems,
03:41
different CPUs, even different NDNS. So it should be easy to port this profiler to all the platforms that we support by Prime Video. Low overhead, I think this is something that all the profilers are trying to achieve. We wanted to measure timings because that's what usually people think of when they do profiling.
04:01
How long does it take to execute a function? But apart from that, we also wanted to measure other metrics like memory usage, CPU usage, how many HTTP calls we've done, and so on. And the other thing that we wanted to achieve, we had a group of people who were very into performance and were responsible for fixing performance issues,
04:21
but we wanted also other developers to use profiling tools on a daily basis. So while they're writing the code, they can focus on performance as well so we don't have to do the job afterwards. So we wanted to provide consistent user experience across different platforms. So no matter what device you're developing on,
04:42
whether this is a streaming stick, web, game console, smart TV, you always have the same tool, so it's easier. They just need to learn one tool and they can use it for all the different projects. So that was a list of features, and we come up with a very simple, high-level design.
05:00
We have a profile device that's running the application. We have a Hulk Tracer library attached to that application. That generates a binary stream of events. We call it htdampstream, and then we have a client which is on the developer desktop that's converting this byte stream to some human readable form like, for example,
05:22
Chrome trace format, flame graphs, or you can, as I'll show you later in the demo, you can build your own client very easily to do some other visualizations that our client doesn't provide. This is a bit more detailed diagram. So there's an app.
05:42
This app has the Hulk Tracer library linked, and Hulk Tracer has a timeline. A timeline is essentially a buffer, and application pushes events to this buffer. These events are serialized, and then once the buffer's full, we push it to a listener, and the listener is basically a function callback
06:03
so that then the listener decides what to do. And Hulk Tracer by default provides two listeners. It's either it stores the stream to a file or it can send it over TCP IP, but you can extend it. I don't know, if you wanna send the stream over a serial port, for example, you can do that. And then the stream goes to the developer desktop,
06:23
and again, we provide the library as written in C++ and has Python bindings that deserializes the stream of events and provides you nice data structures in your language so then you can build your client. And we already provide the Hulk Tracer converter client that converts this binary stream
06:40
to some common trace formats. We also have the library that's written in Rust. It's not really experimental because I know people already use it, and I'm gonna use it today during the demo. So if that's gonna work, then it won't be experimental anymore. Hopefully it will work. So I mentioned here, you see that there's a timeline,
07:03
and the timeline is conceptually as a buffer, but it actually is a little bit more, and you need to provide some configuration for that timeline. And so to simplify this process of creating the timeline, Hulk Tracer provides a global timeline which is quite efficient. It's kind of multiple timelines, actually.
07:20
It's timeline per thread, so if you push events to the timeline, we don't require any locks because you have an instance per thread, so there's no problem with data races and so on. And you can easily access it by just calling ht-global-timeline get function. I recommend just using that. To be honest, I don't think in my real projects
07:41
I ever use different approach, so it's probably good enough. So I mentioned that the basic data unit in Hulk Tracer is an event. So we have quite a few events defined in Hulk Tracer, the event types, but you can define your own. As basically a C structure,
08:00
you define it using a C macro, and you provide the event class name, which could be my event, for example. It supports inheritance, so all the events must eventually inherit from ht-event, which is a very base class, and then you provide a set of fields that you want to have in this event,
08:21
and you define it by three properties. As a type, it's either integer, string, struct, float, double, or pointer, I think. Then you provide the actual C type of that and the name, and that converts to the C structure, and additionally, it generates automatically
08:42
a few helper methods, like, for example, a method for serializing the events. So when you have this event and you want to push it to the byte stream, this macro already will generate a function that does it for you. And then when you already have the event, if you want to push the instance
09:00
of this event type to the timeline, you just call ht-timeline push event macro, you pass the timeline as the first parameter, the second parameter is the name of your event type, and then all the values, all the fields, and that's pretty much it. So I said that before, that we provide the client
09:20
that parses the binary stream, but then I said that you can write your own event type, so you might be wondering how the client knows how to deserialize those new event types, and for that purpose, we divided the event stream into two sub-streams. There's a metadata stream that describes all the types,
09:42
and there's the actual event stream with all the values. And since everything in Hawk Tracer is an event, even the definition of the event is event as well. So we have special events that describe your type, and the first event is just describing the name
10:00
and the number of fields that you have, and then you have new event for each field of this class, so in that case, we'll expect one event for class info event and three events for class field info event, and in class field info event,
10:21
we provide information like the field type, field name, the size of that field, and the data type, whether it is a string, integer, struct, and so on. And this, both streams are serialized as a byte stream,
10:40
so you can see that basically, it's just like 30 or 40 bytes for those events, and then eventually, you have the actual event, and this event has a type, so it's number nine. As you can see, the class ID is nine here, and the info class ID, nine again, so all of them are kind of connected to each other
11:02
by this identifier. And it's important that you first need to send the definition of the event type before you actually send the first event of this type, because otherwise, the parser doesn't know how to parse this event, and yeah. But the Hawk Tracer does everything for you, so it's just...
11:21
Can I ask a question? Yeah, go ahead. If the parser loses sync with the byte stream, is there a way for it to regain sync later? So no, at the moment, it is not possible. I'm working on improvement that will be actually possible, but at the moment, if you lose the metadata stream, yeah, there's a problem. So yeah, but I'm aware of this problem,
11:42
and we'll be working on that. Okay, so I mentioned that we also want to measure time, and that was actually quite important requirement for us, so to make it even easier, we have predefined event types that are already for measuring time.
12:00
They take a label, and they take the duration and the timestamp, and we have macros that automatically generate those events and push them to the timeline. And so for C++, depending on what you want to do, if you just trace the whole function, you just put this macro,
12:20
httrace function at the beginning, and if you just want to trace the random scope, there's httrace. There is a few more macros that are more optimized for that. For example, they have some hash maps, so you don't have to send the string every time for every event. It's even more optimized version that uses some tricks with static thread local objects,
12:43
so even hash map is not necessary for some of them. It's documented, so I don't want to spend much time on that, and we did bindings for a few languages. For now, the public one is Python and Rust. We have one for JavaScript as well.
13:00
It's not ready yet to be published, but we'll do that in the future, and for Lua as well. So this is how you do that in Python. You import the trace decorator, and you just put it for the function that you want to trace, and the same for Rust. There are some macros. I'm not an author of the Rust bindings,
13:20
so probably if you want to know more about this, Alexandru, he's in Fosden. Not sure if he's today in this room, but yeah, he's an author of that, and basically, that's how you do that. Also, there are some kind of Rust macros that you can trace the scope. Okay, so that was it, and I just want to show you some demos,
13:43
so you can see how you can use it. The first one is, we have a C++ application that allocates the memory, and then we want to see on the graph how the memory grows. We also want to know how many allocations we've done in this program. So we'll write a simple Python client that receives
14:02
the Hulk Tracer event stream and does the visualization. I start with the program itself. So at the very beginning, we need to initialize the Hulk Tracer library. So this is the event type that we define. As I said, there's a name, the base type,
14:22
and we have two fields, memory usage and allocation count. They both are integers, size T type. The event, before you want to use it, you need to register it. This method is auto-generated by the macro that I showed before, by this macro, htdeclareEventType.
14:42
You just need to call it before you use it. It registers some information in Hulk Tracer system. Then you create the listener to the timeline. I said the global timeline is the best option. So we're using global timeline here. There are some parameters like port. This is the buffer size. We set it to zero, so we don't actually buffer anything.
15:03
We stream it directly. To the client, and we say that we want to use TCP clients so we can stream it directly to the client and we can get the real data. Hulk Tracer already provides the functions to get the memory usage. It also provides some functions to trace the allocation.
15:24
So you just need to provide a callback. This is for pre-malloc, then it's post-malloc, pre-realloc, post-realloc, and so on and so on. We are only interested in pre-malloc hooks, so we register that. And this is basically our function.
15:42
We get the memory usage. We get the allocation count, which is read in the other callback. So here, this is our malloc callback. So we increase the allocation count every time somebody calls malloc. So we get that all in the context object.
16:01
And that's basically what we do. We push the event. So this is our type. Those are our values. And this is the memory usage, virtual memory usage, and this is the allocation count. And this is the client. It's written in Python. All we need to do, we need to start the client,
16:22
the Hulk Tracer client. We say, okay, listen to 8765 port on this IP address. This is the animate method. That's basically the drawing the graph. But the most interesting bit is this one. So this is basically how we read events from Hulk Tracer.
16:42
This is our client. So we're waiting for end of stream. If it's not, then we pull the event. If there is an event in the queue, we check the first, this is a tuple. And the first element of the tuple is the event name. So we check if this is the event that we want. And if so, we get the timestamp,
17:02
we get the allocation count, and we get the memory usage. And then we put the memory usage on the graph, and we print the allocation count. So it's very, very simple. Let's run it. So we run the client, and now we run the program that we wanna trace.
17:22
So you can see that the memory usage is growing, and you can also see the allocation count is going up. And so you might be wondering why the memory usage is growing. If we go back to our application, we see that we have a loop, and we do malloc.
17:43
And it's called 100,000 times. So we can see how many mallocs we actually got. It's more than 100,000. There's probably some other allocations going on. Yeah, and this is the graph.
18:01
So that's how you can get the real data from the Hulk Tracer by defining your own event. It doesn't have to be memory usage. You, for example, can trace a number of HTTP calls, for example, and so on. Yeah, so that was the first demo. And the second one is going to be more complex, a little. I wanna show you how you can trace
18:22
multiple languages at the same time, because that was actually our real use case. We had application written in C++, and then we run Lua and JavaScript on top of that. So I do slightly different example. I have a main application written in Rust. This application downloads the image using a downloader library that's written in C.
18:42
It uses cURL. And then I rotate this image using some image. I think it's OpenCV. Maybe not OpenCV. I actually changed it. But yeah, it rotates the image in Python. So I run Python interpreter in Rust. And then at the same time, while I'm rotating the image, I also upload the image that I downloaded
19:02
in the first step to the S3 backend. And I wanna know how much time I spend in each particular operation. So I, again, start with the code. So this is our Rust client. Even though you might not be familiar with Rust, I think it's pretty simple.
19:22
So we start with download file. This is download file RS. It's basically just the wrapper for the download file library function from the library that's written in C. Then once we have the file downloaded, we spawn two threads.
19:41
One is doing the rotation in Python. It starts the Python interpreter and does the rotation. And the other one at the same time uploads the file to S3. You might see that all of the functions here are decorated with those macros. And we also have some other trace points in here.
20:01
We can look at the downloader file. It's pretty simple one. It's just the curl wrapped around. We trace the function here at the beginning. And we also see how long does it take to call curl easy perform. So we trace the whole scope here. And just last bit is the Python one.
20:23
We import the decorator and we decorate all the functions. So we basically rotate the image and save the image. We load the image, we rotate the image, and then we save the image to the file. So not really a rocket science here.
20:41
It's important that if you want to run it with Python, this environmental variable needs to be set. Otherwise, the Python bypasses all the trace decorators. So it doesn't have much impact on the performance if you disable that. So you can even have it in the production code. Okay, so let's run this.
21:04
So this is our program. It downloaded the file successfully. It's doing the rotation and it's doing the upload. So you can see that it actually did rotate some image. Let's start from zero. Yeah, so it's a hook and it's basically all the angles are here.
21:23
If we delete that one, we have also rotate HT dump, which is the binary file with all the events, all the traces. Oh, this is not gonna work because I forgot this environment variable here.
21:45
So we wouldn't have the Python traces. So I run it again. I'm gonna delete the images. So we have the HT dump file, which is the binary stream. We can now use a hook tracer converter.
22:02
This is the one that's experimental. So hub is working. And we generate the Chrome trace format. So we say it will be output.json. Okay, it converted. So now you can use the tool that's deployed
22:20
with Chromium. And you see that, first you see that we have three threads here, one, two, three. And that's exactly what we had because we had the main thread and then we spawned two extra threads for rotate and for upload. This is the first bit that's in C++ or C.
22:40
We download the file, this HTTP request. And then just after that, we start rotating Python. And those trace points here, those are from Python. Save image, rotate, main. Those trace points are from Python. And this one uploads to S3 is in Rust. So as you can see, you can trace. You can have everything in one trace file
23:03
and you can easily analyze what's going on in the program. You can also convert it to flame graphs. So if you want the flame graph view,
23:22
you can just do that. This is a very simple program. So the flame graph also is very simple, but yeah. And so that's it. We have some plans for improving this. There are some of the items, but not all of them, as you mentioned. We wanna add some extra protection layer in case we lose some events on the way.
23:44
And yeah, that's it. Thank you. There's a bunch of links. Feel free to contact me. There's a documentation on the website. There are tutorials. They say how to integrate it with your project. Basically, Hawk Tracer can be combined into one single C++ and header file.
24:00
So it's easy to link it to your project if you want. There are other links for bindings and for converter. So thank you. I think we'll have one minute for questions, but if that's not enough, then I'll be around.
24:23
Okay, yeah. Which methods, like you showed TCP now, right? So in addition to TCP, what do you currently natively? So this is the TCP that we use in the first demo and the file which was used at the second demo. Yeah, so those are natively supported at the moment.
24:42
Do you want something like serial? Sorry? So if you want something like serial? Yeah, you need to write them in your own. Sorry, the question was what listener types we support and my answer is currently we support file and TCP listeners, yeah. Thank you.