Linux tracing with LTTng
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32566 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2014165 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Projective planeTrailRight angleResultantEvent horizonComponent-based software engineeringXMLUMLLecture/Conference
00:50
Network topologyEvent horizonMathematicsTracing (software)Term (mathematics)Model theoryComponent-based software engineeringContingency tableDifferent (Kate Ryan album)SpacetimeModule (mathematics)Kernel (computing)Server (computing)Run time (program lifecycle phase)Procedural programmingMereologyBinary filePresentation of a groupMiniDiscFile formatInstance (computer science)BefehlsprozessorDistribution (mathematics)Shift operatorWordMultiplication signRoundness (object)Structural loadFile viewerBinary codeSystem callBit rate
04:08
Patch (Unix)Run time (program lifecycle phase)Tracing (software)Video gameEvent horizonNetwork topologyCompilerLibrary (computing)Process (computing)Cartesian coordinate systemDemonModule (mathematics)Game controllerPredictabilitySlide ruleSpacetimeLine (geometry)Instance (computer science)Windows RegistryFreezingStreaming mediaSoftwareKernel (computing)Utility softwareVirtual machineLecture/Conference
05:26
SoftwareComponent-based software engineeringTracing (software)DemonProcess (computing)Point (geometry)Slide ruleBefehlsprozessorCartesian coordinate systemPhysical systemVery-high-bit-rate digital subscriber lineDifferent (Kate Ryan album)InformationKernel (computing)Event horizonComputer filePublic domainAdditionOrientation (vector space)Virtual machineNumbering schemeLibrary (computing)Video gameInstance (computer science)SpacetimeoutputLecture/ConferenceXML
07:59
Cartesian coordinate systemDemonVirtual machineVideo gameInsertion lossCrash (computing)Buffer solutionSpacetimeDifferent (Kate Ryan album)Network topologyWorkstation <Musikinstrument>Euler anglesIncidence algebraAdditionLecture/ConferenceUML
08:50
Shared memoryArithmetic meanOpen sourceSpacetimeSemiconductor memoryLibrary (computing)Data structureMechanism designSynchronizationSystem callLecture/ConferenceUML
09:30
SpacetimeBefehlsprozessorQuicksortScaling (geometry)Speech synthesisTracing (software)Series (mathematics)Point (geometry)CodeSoftware testingExecution unitRun time (program lifecycle phase)DigitizingField (computer science)Network topologyNumberRight angleProof theoryElement (mathematics)BenchmarkAreaMacro (computer science)Physical systemEvent horizonMultiplication signInformation securityInternet service providerInstance (computer science)Function (mathematics)Pointer (computer programming)Data structureString (computer science)BuildingState of matterEscape characterCartesian coordinate systemTerm (mathematics)SummierbarkeitFlagAffine spaceProcess (computing)Kernel (computing)Link (knot theory)WeightParameter (computer programming)Message passingLetterpress printingFrequencyClique-widthTwitterBuffer solutionFlow separationAdditionSpring (hydrology)Functional (mathematics)Streaming mediaPublic domainAsynchronous Transfer ModeComputer fileCountingComputer wormScheduling (computing)Scripting languageView (database)Type theorySystem callRing (mathematics)2 (number)Dirac delta functionTimestampLecture/ConferenceComputer animation
16:28
Kernel (computing)SpacetimeTimestampSoftwarePublic domainLocal ringVirtual machineComputer fileData storage deviceMereologyPoint (geometry)Real numberCartesian coordinate systemInstance (computer science)Goodness of fitTracing (software)Lecture/Conference
17:10
Cartesian coordinate systemRow (database)StatuteProduct (business)Kernel (computing)Core dumpCycle (graph theory)ImplementationInstance (computer science)Set (mathematics)CASE <Informatik>Real numberXML
18:03
Event horizonObject (grammar)Buffer solutionCartesian coordinate systemNetwork topologyRight anglePattern languageGroup actionData loggerIntrusion detection systemDifferent (Kate Ryan album)Scripting languageRepository (publishing)Core dumpCross-correlationReal-time operating systemVirtual machineLecture/ConferenceMeeting/InterviewXML
19:11
Multiplication signTerm (mathematics)Instance (computer science)LoginComponent-based software engineeringReverse engineeringProcess (computing)HypermediaProfil (magazine)Cross-correlationPlug-in (computing)Lecture/Conference
19:48
Drop (liquid)Instance (computer science)HypermediaSynchronizationEvent horizonStaff (military)MathematicsTracing (software)Cartesian coordinate systemDifferent (Kate Ryan album)Term (mathematics)XML
20:42
Kernel (computing)Cartesian coordinate systemBuffer solutionWindows RegistryMathematicsQuicksortPhysical systemForcing (mathematics)Different (Kate Ryan album)Series (mathematics)Boss CorporationComputer fileWordLecture/Conference
21:26
SpacetimeContrast (vision)Server (computing)Level (video gaming)File viewerTracing (software)Virtual machineDemonMedical imagingPhysical systemReal numberSoftwareArray data structureSystem administratorWorkstation <Musikinstrument>Kernel (computing)Block (periodic table)Video gameInternet forumInstance (computer science)Program flowchart
22:49
MassScaling (geometry)Virtual machinePrisoner's dilemmaElectronic program guideNetwork topologyModule (mathematics)RootCartesian coordinate systemKernel (computing)DemonLaptopLecture/ConferenceComputer animation
23:58
Musical ensembleNetwork topologyRandomizationDecision theoryKernel (computing)FreezingArithmetic meanImage registrationRootDemonCASE <Informatik>Public domainEvent horizonElectronic mailing listLecture/ConferenceComputer animation
24:36
RootMusical ensembleRight angleLaptopOnline helpEvent horizonKernel (computing)Different (Kate Ryan album)Factory (trading post)Standard deviationInsertion lossXMLLecture/ConferenceSource codeComputer animation
25:23
Kernel (computing)NeuroinformatikEvent horizonMathematicsPoint (geometry)Ocean currentComputer animation
26:05
Cartesian coordinate systemEvent horizonDifferent (Kate Ryan album)SpacetimeSemiconductor memoryKernel (computing)Right angleDefault (computer science)Function (mathematics)BefehlsprozessorNetwork topologySoftwareSynchronizationComputer fileLatent heatParameter (computer programming)BitData storage deviceFile viewerMathematicsComputer animationJSONXMLUMLSource code
28:07
Event horizonVirtual machineMiniDiscSystem callTracing (software)BefehlsprozessorLine (geometry)Different (Kate Ryan album)2 (number)Row (database)Scaling (geometry)Insertion lossThread (computing)Cartesian coordinate systemMultiplication signOperating systemBuffer solutionVery-high-bit-rate digital subscriber lineCurvatureoutputContent (media)BitStreaming mediaNetwork topologyWave packetDegree (graph theory)Point (geometry)SoftwareOffice suiteFreezingMathematicsType theoryTwitterLatent heatResultantMusical ensembleRight angleIdeal (ethics)Diagram
31:12
Java appletRevision controlFreezingPoint (geometry)Alphabet (computer science)ResultantMedical imagingMereologyLecture/ConferenceXML
32:23
Library (computing)Design by contractQuicksortCartesian coordinate systemJava appletTracing (software)Combinational logicProjective planeProcess (computing)Kernel (computing)Social classRight angleMultiplication signMilitary baseDynamical systemOpen sourceInternet forumNetwork topologySoftwareLine (geometry)Interpreter (computing)Computer hardwareArmTimestampSpacetimeXMLComputer animation
34:49
Java appletKernel (computing)Interface (computing)Term (mathematics)Tracing (software)PlanningFeedbackInterpreter (computing)Heegaard splittingKeyboard shortcutPoint (geometry)InformationComputer programAddress spaceMechanism designEvent horizonFunctional (mathematics)Absolute valueSystem callProjective planeGraphical user interfaceSpacetimeCodePatch (Unix)Cartesian coordinate systemOpen setReal-time operating systemMoment (mathematics)Different (Kate Ryan album)Physical systemInstance (computer science)Semiconductor memoryVirtual realityElectric generatorModule (mathematics)Multiplication signParallel portOpen sourceMiniDiscRight angleSign (mathematics)Closed setRepository (publishing)Arithmetic meanLatent heatElectronic program guideFilter <Stochastik>Asynchronous Transfer ModeVirtual machineMathematical analysisSymbol tableVirtualizationHookingWordMereologyMathematicsAdditionResultantPower (physics)Network topologyVisualization (computer graphics)Integrated development environmentProcess (computing)Food energyInformation securityLine (geometry)Standard deviationNeuroinformatikClient (computing)Contingency tableDisk read-and-write headBinary treeRule of inferencePattern languageInferenceMedical imagingCanonical ensembleCondition numberTheoryConcentricMaxima and minimaProfil (magazine)DialectContext awarenessDigitizingRevision controlService (economics)Dynamical systemOracleBitIntegerTable (information)AreaCore dumpPublic domainQuicksortLecture/Conference
Transcript: English(auto-generated)
00:00
Hello everyone, can you hear me? Fine, great. So my name is David, David Goulet, I work at the LTTNG project. Also I work at Efficials, which is a company that basically we make features and development in LTTNG. So today, so usually tracing talks are kind of a bit boring, but I try to do my best to make it fun.
00:25
So just to show you, who ever heard about LTTNG? Oh that's great, at least two people, great. So tracing in general, can just show you who uses tracing with S-Trace perf, S-Trace also.
00:45
Alright, cool, so I'm on a good track. So I maintain the LTTNG tools component of the LTTNG toolchain. And this presentation is about tracing with LTTNG, but showing you what we did in the last three
01:03
years in terms of excellent feature and great work and how you can use it in user space also. So that's really a quick overview of LTTNG and tracing in general. I said probably people here maybe don't know what tracing is, so I'm just going to do an overview about tracing.
01:22
Then we're going to talk about everything else and some future work. So the difference between tracing and debugging or having logging with syslog and so on, is that you can enable or disable events during round time. And tracing is low intrusiveness, very high performance.
01:43
So with LTTNG we created this user space tracer and a kernel tracer and then we created the LTTNG tools component that mixed them together. So you can merge kernel and user space trace within one trace and analyze that after all.
02:01
So tracing is basically eye throughput, eye performance debugging with like, let's say you have printf and then tracing uses that printf but eye performance rate and it's scaled over CPU's also. So this is basically what tracing is. So for instance strace, so everyone knows strace here, the great tool right?
02:22
It uses ptrace and just to hijack and understand how to print out the syscalls. So tracing is a bit different from debugging, it's very important to understand because this is basically what this talk is about, is to make you use tracing in the end.
02:45
So LTTNG 2.x, so we have 2.4 I think right now, is unified tracing. So unified tracing between user space and kernel space. As I said, low overhead, we use a common trace format called CTF which is a tracing format,
03:03
it's a binary format on disk and then we analyze that with viewers, whatever viewers you can use. And we are now shipped in Ubuntu, Debian, Red Hat and other distribution. So as I said, so there's two tracers where you have the LTTNG modules which are the kernel trace.
03:23
So as a, I'm going to repeat that like a lot, you don't need to recompile the kernel with LTTNG, you use modules as of now. So the difference between LTTNG, between perf and LTTNG is that perf is in kernel. They do a lot of clever stuff, they do also profiling where LTTNG doesn't do that.
03:46
But with the kernel module, with the kernel tracer, you can trace a lot of events in the kernel. Sys calls and a bunch load of whatever happens in the kernel in different subsystems. And then we created this USD, this is what I'm mostly going to talk about today, is this user space tracer.
04:05
So the love of development without printf, so we're going to use tracing in user space. So this tracer is an in-process library, so you're gonna, you instrument your application, you recompile and you have your tracer. But the good thing is that you can enable, disable during runtime any events, you
04:24
can live trace it, you can snapshot, there's a lot of features, it's pretty fun. Yes, so you can use the kernel as in 2.6.38, before that there's three patches to the kernel, but after that there's no patches, only modules.
04:42
The utilities of LTTNG, so LTTNG is, we created a daemon, which is called a session daemon. The LTTNG session D, the third one, is the tracing registry daemon. And this thing will combine everything you do, so when you create a trace, for instance, so you
05:00
do an LTTNG create with a command line, you can use it for kernel and user space tracing. So this is pretty great where you can integrate both sides into a trace. So yes, we created a control library, you can use it, it's LTC, so it's a C library, control library, we have a relay D, which this relay D makes, you can stream the trace over the network,
05:25
so you can extract traces from machines and put it on the network and analyze it later on, so this is the relay D, the streaming daemon. And the consumer D is the one we extract, so there's a bunch of daemon, you don't really need to care about that when you trace with LTTNG,
05:45
but if you want to contribute or make development in LTTNG, those are kind of the component. The viewers, we have a Babel trace, which basically takes an input, a trace input, a directory, so the traces and the file traces, and just print out on the console, human readable.
06:05
There's LTTNG top, and this is pretty awesome, so everyone here knows top, or each top, and this LTTNG top is exactly the same, but it uses kernel traces to print out the information on the top.
06:22
So the difference between top and LTTNG top is that we don't probe proc every second, for instance, we just use live traces, and you can have your output of what's going on in your system. This is pretty awesome, because usually the performance are one to three percent or five percent it on your system, where top takes a lot of CPUs.
06:47
Alright, so this is LTTNG user space, you instrument your application, I think I have a couple of slides where it shows what's an instrumentation, what's means that,
07:00
we call it trace points, so you put trace points in your application and you can extract at high speed your data. So in process library, the session setup, you run your app, you collect your traces, and then you analyze, that's it, that's all. So this is a quick schema of what's going on, so you're in a machine, and then you have your
07:26
session daemon, the session daemon as I said is a tracing registry, it just helps you create sessions and enable events. So on the command line you do let tng create, enable event, dash u means dash user space, or dash k dash kernel, and
07:43
we have other, we call that domains, we have other domains that I'm going to show you later on, and dash a means all. So you enable all event of all your application, so with LTTNG the great thing about that is, if you have instrumented application, you can list which application you can trace, so if you start your application on your machine, and they're going to register to that session daemon, and so the session daemon
08:07
will respond to consumer and say, alright, okay, I'm going to consume your trace, I'm going to extract your trace, and that's it. It's pretty easy. And the instrumented application, so our consumer will create buffers, and will share it with the application.
08:25
So the application will just populate those buffers with the tracing data, and this is the great thing about tracing user space with LTTNG is that, for instance, when your application crash, well, the buffers are still alive. So we have the buffers, we have everything to consume them, we know the application is dead, so
08:45
we just extract the data, and then you get all your trace, even though the application is dead. And this way is done with a shared memory. So we use, we are using heavily user space RCU, so anyone heard about RCU here, show of hands? RCU, yeah right.
09:06
So RCU is a very complicated thing, but it's a user space RCU, RCU means read, copy, update. It's a synchronization mechanism to have very, very high efficient and very high performance data structure, a lot less data structure.
09:26
So in user space, there's a library, RCU library called user space RCU, you can check it out. And we use that a lot just to synchronize our trace and our performance, because as I said, if you have 8 CPUs or 128 CPUs, it's going to scale over that.
09:46
So let's, oh sorry, so yes, this is basically how we do stuff in user space. Shared memory, we exchange buffers, and that's it. Alright, cool stuff, alright cool.
10:02
So this is how you instrument an application. Yes, it is C, it's macros, it's kind of complicated to be honest. We have a helper script that helps to create that, but in the end it's kind of a tedious process.
10:25
But again, the usefulness of tracing is that you instrument your application and you do it once, you don't put printf everywhere. You just instrument, put on trace points, and then you can trace it over time.
10:42
So your application is running, and at some point there's a problem. Well, you can just enable a couple of events that you think is the subsystem is badly working, and you can extract your data. So this is a test if you go into the LTTNG repository, we have examples.
11:06
And this one is a URL example. So your provider name, trace point name, those are two different names just to separate the trace points. So you can have a bunch of trace points in your application and just separate them into providers, or domain, or whatever.
11:22
So instance, your company name, and then the department, and then the application, then the subsystem, and that's it. You have your provider name. And then you can just trace the fields, the payload you want. So you record the ints, strings, structure, whatever.
11:42
So we have a very basic function here, and then this trace point that we created here. So UST test hello, and then we're using it, right? So when you're coding your application, or maintaining it, or whatever, you can add those trace points to the code,
12:04
and you only put trace point, and then the fields, and it's going to record. So if the tracing is not enabled, there is absolutely no performance it. So the trace point is going to be it, it just zeroed out, and that's it.
12:23
If you enable it during run time, and then you hit it, I think I have a benchmark performance chart, but as of now we are around 250 nanoseconds it takes to your application to hit the trace point. And it's kind of the lowest we can go right now.
12:41
There's no syscall in the path, it's all zero copy stuff, so this is very, very high performance. So it's pretty high, pretty cool stuff. And yeah, trace point. Tracing session, so you come in line, you create a session, so let's say you want to enable the subsystem number one and the subsystem number 42,
13:04
you can use wildcards like that, saying just subsystem 42 underscore, and then every event in that subsystem will be enabled. So dash U, user space, and then the subsystem name. And you can start, and the LTTNG start means it's going to start tracing. The trace is going to get recorded.
13:22
And then you can wait, you can do whatever you want, get coffee, and then you stop at some point, and then you can view it. So the view is, so this is an example of the trace you can extract. So you have the timestamp on the top, top left,
13:42
and then you have the delta between the two events, the orange thing is the deltas between the two events. So one second between those two. Then the host name, the trace point name, provider and trace point name, and then the payload. And you have the CPU ID saying this event was done on that CPU, CPU one, CPU two, CPU three, whatever.
14:02
And then the payload. As I said before, the payload, you can put whatever you want. Anything, strings, int, structure pointers, you can extract everything. So this is a human readable output made from Babel trace. For a trace.
14:21
So this is another example of the kernel. So for instance, this is called read, this is called open, and a sketch switch. A sketch switch is when two processes get switched on the scheduler and the kernel.
14:40
So yes, for instance, read, you have the payload read, the FD, the buffer pointer and the count. Open, you have the file name, the flags, and so on and so forth. Perf gives out around the same output in terms of kernel tracing. It's pretty similar to S trace also. S trace can understand the layout of the Cisco, for instance,
15:07
if you do receive message or send message. The payload is going to be, you're going to know the type of the payload, which means the structure, the arguments and so on.
15:20
So this is why buffer is a pointer. We don't know if, what points or what's the structure in this. It's just a pointer. But again, human readable. So again, start, stop, link and view. But this is not very useful in the end because you just start a trace, you trace and then you stop. So we created snapshots.
15:43
Snapshot means you're running your application in flight recorder. It's ring buffers, so you trace data, you get your data, you put it in the buffers, and at some point in time, depending on the size of the buffers, it's going to get, sorry, it's going to get erased by new data.
16:06
So right, ring buffer, flight recorder. The point of that is that, let's say you monitor, you have your instrument, your application with UST, with trace data, with trace points. And then you start, you just enable everything and then you start it. You just forget about it because it goes into flight recorder mode, ring buffer mode.
16:24
And you have some Cacti or Nagios alert and you trigger a snapshot. So what this does is that you're going to record a snapshot, you get everything from that point on when you take your snapshot. And it's going to extract your data, put it on disk, or put it on network. As you choose, you can put it on network, it's going to send it to a real AD,
16:43
LTTNG real AD, it's going to store it on file on another machine or on your local machine. And you can then analyze your trace. The good thing with LTTNG and why I use LTTNG is that you can do that with the kernel and user space. So you can snapshot both domain, both tracers, and you can merge them
17:01
because the timestamps are asynchronous. So you have your trace data from your application and from your kernel. So this is pretty great because you can use that to monitor an application that runs in production, for instance. So yeah, create, enable, start, and you wait, you do something, and you snapshot record.
17:22
And that's it. So real world use cases where we use that and it's still being used. Where we know a large set up that are being used with that is, for instance, we have a core handler.
17:44
A core dump handler. Where you have a core dump on the application, segfolds, whatever, anything. Your application core dumps. Well, you can tell it to, when it core dumps, it snapshots. And this is really, really useful, especially with the kernel, because you can know exactly what happened with your application and the kernel
18:02
when this application core dumped. And as I said, the buffers are shared with the consumer. It means that when it core dumps, you have the assurance that you're going to get every event that will be triggered by the application. So we have that in the tree right now. So if you go to git.ltng.org, you have the repository there,
18:22
and we have this fancy bash script that allows you to just snapshot when it core dumps. IDS also. So, let's say you are tracing regularly,
18:41
you're just tracing your application in your kernels, on different machines, different nodes. There's right now research being done at the University in Montreal, where it uses the data, live data, that is uploaded from lttng to create IDS to analyze that on the spot, real-time,
19:02
and just trigger action if some pattern emerges. So lttng is being used to correlate data logs to merge with syslog also. We try to work with plugins that allows you to use tracing data
19:21
and put it with, for instance, Splunk or syslog in terms of time correlation. You can use performance profiling, so on and so forth. So, as of last year, I think September or something like that, we are still in the release candidate process for the 2.4 version,
19:45
but we added the live component. So the live component means you can trace your data, you can trace application kernel, and you can read the event as they're being recorded. That seems easy to do, but it's kind of very difficult,
20:00
so in terms of synchronization and performance, to keep that performance on, but after a lot of work, we added this feature upstream. So as of now, the lttng tab, for instance, uses that. You create your trace, your session, your trace, your kernel, your trace, your application, and then you can live read your trace.
20:23
So, one of the things we're trying to do is to have a drop-in replacement for strace, for instance, where you can use lttng to do the same thing as strace does. And the difference between strace and lttng is that strace changes your application.
20:41
It uses ptrace, it changes stuff, and the behavior of your application within strace and without strace is different, because ptrace is very, very intrusive, and changes registry in doing some jump, trampoline buffers to extract this data.
21:02
So, imagine you can use a kernel with a 3% hit performance on your system, and have the exact same thing with strace. This is what we're trying to do. This is why we come here to Fuzzden to just explain this fantastic feature and not sell it to you, but also try to make you contribute and help us
21:20
to develop and get contributors in lttng. So, this is the kind of stuff we're working on. So, the live tracing, the thing is that you can contrast analyzes, cluster level analyzes. You can do a lot of things with live, you can imagine, where you extract data live. And one other thing with lttng in the last year is that we try to bring tracing to not kernel developers.
21:46
So, to user space developers, to system administrator, just to use tracing, not in a development fashion, but in a monitoring and useful way. So, this is a quick image of the infrastructure of lttng,
22:04
where you have lttng session D on your host. So, this means lttng session D is the daemon that traces. So, the traces are being collected on the lttng session D on the top server A, server B, server C. Those servers are being traced. And you can extract those through TCP, to real AD,
22:23
and then you can have your viewer connected to that real AD over TCP. So, for instance, we work with companies, large companies like Ericsson in Sweden, that uses this kind of infrastructure, where they have a huge amount of servers,
22:40
they extract the data to a couple of other machine through network, and then they view it with other machine to analyze it through the network again. It scales on a massive amount of machines, and it's being used. So, this is not just like, you know, just some developers having a dream.
23:06
It's being used. So, yes, I wanted to demo you lttng stuff, but I kind of have a problem with my laptop this morning, so I can't, unfortunately.
23:21
But what I can do, though, is just to show you some basic lttng stuff. So, what we have is lttng, right? So, lttng, let me see something, that will be four.
23:46
So, one thing is that if you have a session daemon running as root, it's going to load automatically the kernel module, the kernel tracer. If your user is in the tracing group, so we added this tracing group,
24:01
you can change the name, I mean, it's just comfortable, but the tracing group allows you to talk to the session daemon, and the session daemon is running as root and can trace the kernel, so a normal user can trace the kernel. So, I'm in that tracing group, so I'm going to lttng create a session, there we go.
24:20
So, there's no session daemon, so it's spawned, and then it's going to write the trace there. So, lttng list-k, that means list the event of the kernel domain. Yes, I need the, I'm not in the tracing group. All right, okay. Let me put you on with the root.
24:43
Sorry, my laptop is a mess right now. All right. Okay, put it back on the root stuff. There we go. Okay, lttng list-k. So, this is all the kernel event you can trace. You see, xtfs, btrfs, xt3, rq, kvm.
25:05
So, those are standard events that are in the kernel right now. They're called trace events in the kernel. So, perf, I have the same events. Ftrace is also, I think ftrace is listing those events also. And then you can trace whatever you want.
25:21
So, let's, oh yeah, okay. Let's trace them all. And I'm going to see how much events can you create. So, I'm going to say just enable all events in my session for the kernel. Oh, I need to create a session, though. Sorry. All right, enable event, and I'll show you this, okay.
25:42
And it is, so all the events are enabled, and there's a lot right now. So, when I do lttng start, so tracing is starting. So, at this point, my computer is tracing the kernel. Just to show you how much data it's going to create.
26:03
I think it's, h. So, let me just try to center that. Oh, we can't see, right? 11 megabytes, and then 12 megabytes. So, right now it's tracing. So, those are the files, the data files.
26:24
So, channel zero means channel zero is the default channel. So, you create a channel, and you put on events in those channels. So, you can have different events in different channels, in different sessions. You can move that around in the network or not. So, as of now it's tracing, and I think my CPU is about doing nothing right now.
26:43
But I'm extracting every, every event in the kernel right now. It's a lot of data, and there's nothing. There's Tor in awesome, that's it. So, let's stop that, and I'm going to show you how it looks like.
27:02
Yes. Oh, yeah. It'll be configuring. There you go. So, here's the data, and it's non-stop. I have an output class, and I'll show you how it works. Non-stop. All right. So, this data is being recorded.
27:21
This is kernel's event. Every KMM allocation, every memory allocation, every sketch switch, everything. Every RQ, everything is being traced. So, let's try to stop that. And just to show you the amount of data you can extract without any performance yet. And this is kind of really great.
27:41
And you can, as I said, you can couple that with any user space application. So, if I have any application, lttng list-u, and you can see the application, and what you can trace when the application starts. So, the application registers the session daemon, and then lists the events.
28:00
So, you can just list-u, see what application you can trace, and start tracing. All right. So, let's go back to... All right. So, it is just some performance results. Just to show you the awesomeness of it. So, we run that for 50 minutes.
28:22
And we took a snapshot. Every, I don't, every 30 seconds. And it created a snapshot of seven megabytes of data. And the old strace, so we compared it to strace. So, strace would create about 5.4 gigabytes of data in a matter of 50 minutes.
28:45
With 61 million events. So, remember strace is syscalls for a specific application. lttng will trace every syscall that happens in your machine for every application. And not only syscalls, but all the other events you saw.
29:01
It creates 6.8 gigabytes, 250 million events at the same time. So, those are the same. And we have a 1% event loss. So, event loss is basically that there's way too much data at high speed. So, we have to lose events, unfortunately, to be able to have this kind of performance and record.
29:28
So, this is a dedicated disk for trace. So, it means that we have a disk and the trace is being put on that disk. A dedicated disk, not a shared disk with your operating system.
29:42
And you can see there's five lines, but you see two right now. So, the red one is strace MySQL. And the rest is the same with lttng. So, there's no performance there. So, it's the same as node tracing. And we are comparing that with flight recorder, as I said, where we're just recording, recording, recording the buffers.
30:07
And the yellow one is the streaming. So, you extract your data, you stream it on the network, so you extract them to another machine. And there's absolutely no performance yet, but you trace 250 million events.
30:21
So, in a shared disk, it goes a bit more badly. Because then you have very high contention on the disk and IOs are off the roof. So, node tracing, of course, high performance. And then you get your strace, which is the yellow one, and lttng, just scalable.
30:41
So, this is why you can see the flat line of strace. Is that when you go to 32 threads on a 32cp machine, then it doesn't scale at some point. It stops scaling. And it just doesn't go linearly where lttng does. The difference is that we have per CPU buffers.
31:00
And we try very hard to just avoid cache line bouncing and a lot of performance problems we have when you have multi-trading application. So, oh yeah, 30 minutes in my talk. Recent work in new features. So, lttng 2.x has been created in 2010, I think.
31:26
And each version is the name of a beer in Quebec, Montreal, Quebec, in Canada. So, we have microbreweries in Quebec. We have very good breweries. So, we're following the alphabet, so we start A to whatever we are.
31:43
So, 2.0 is Annandale, 2.1 is Basmes, 2.2 is Couda, which is a very nice IP from a bar microbrewery in Montreal. So, 2.4 is Épaca pack, the E one, which is a microbrewery in Quebec.
32:02
And 2.5, I don't remember what the name is going to be, but it's going to start with F. So, in 2.4, we added live tracing, as I said. We have also a snapshot that's been done in 2.3, but you can use them in 2.4 or in 2.3, whatever.
32:21
And we added also Java useful support. So, I don't know if there's a lot of Java developer here. I'm not really a Java developer. I don't really like Java, but there's a lot of people using Java. So, we have this contract to support, not support, but add a feature of using Java user logging, so GUL.
32:43
Java user logging is a library where you log your application in Java. So, we hooked, we have a LTTNG agent in Java that hooks itself on the Java user library and extract every data from that and put it on an LTTNG trace.
33:02
So, this is pretty nice because if you're doing Java, you can do nothing to your application. One line, like just Java LTTNG dot create your class, or I don't remember exactly what it is, but just you create the class and then you start the LTTNG thing. And you can use LTTNG as I did with the console and trace your Java application.
33:24
So, yeah, so there's a performance it, but again, it's pretty useful. Future work, I'm almost done, so there's a lot of fun for questioning after that, but future work is that we're going to add this year hardware tracing. We've been talking a lot with ARM people, embedded people,
33:41
and we're going to try to put on the hardware traces into LTTNG. So, once we can do that, it means that you can combine hardware traces, kernel traces, and user space traces in the same trace. You can correlate the timestamp and then analyze that, which is, to be honest, pretty awesome.
34:02
And you can do that right now unless you use some proprietary software from Microsoft. Or Oracle is trying to close source the trace, but it's coming up. And then trace triggers, Android ports, a port for Android,
34:22
and then automatic analysis, dynamic also instrumentation, where D-trace does and we don't unfortunately, but we're trying on that. So, if you're looking for an open source project to contribute or even a job, dynamic instrumentation is very important for us, and we're trying to do that.
34:41
Yes, so LTTNG project, that's an overview of it, so if there's questions, that's the right time. Yes, I think there's a microphone around. There's one person over there. Right there, over.
35:00
So, just one thing for the question and answer. Please sign clearly if you have got a question, so we can come around to your place to give you the microphone. And if you want to leave, please leave, so the others have the chance to ask their questions.
35:21
So, you mentioned that there are standard tracing points in the Linux kernel, so that I don't have to recompile. And as far as I could understand, you can log to the same machine, for example, on the disk, right? You can what? I can trace what?
35:43
So, I enable everything in the kernel, and then I want to log this data on the local disk. It happens in parallel. So, do I get spurious log entries in my trace data?
36:02
So, you're saying that while we're extracting, do we get those events in the data of LTTNG working? So, yes, there's a kind of a feedback loop in there, where you see the LTTNG events.
36:20
So, we're extracting data, we're using, of course, syscalls, and you see them in the trace, yes. So, the more you trace, the more you create data. But you can filter that out. We have filters in the UST, and then it's going to come with the kernel. So, you can just remove LTTNG even. That was the question? Another question. Can I inject these trace points dynamically into the running kernel,
36:45
and do I want to do this if yes? So, the question is, can I dynamically add trace points to the kernel, right? So, the trace events in the kernel are static. So, it means they're coded, they're upstream right now, and you use them.
37:02
What's existing in the kernel is called kprobe, or function tracer, the ftrace tracer. So, you can hook to any symbol in the kernel with that, and LTTNG supports that. So, for every event, you can just add, let's say I want to trace that function in the kernel. So, you can do that with LTTNG,
37:20
with the editable event dash dash function, or dash dash kprobe. So, those are mechanisms that exist in the kernel. You can dynamically trace any symbol, if you have the address or the name. Yes, you can do that with LTTNG. Hi, I have some questions for user space tracing.
37:45
Is there any documentation anywhere for people who are not familiar with kernel programming to do user space probing somewhere? Because when I was looking for something on LTTNG, I didn't find much information. Let's say I want to add some tracing points for Python interpreter.
38:01
Is there anywhere I can look at? So, documentation is always a problem in open source projects, and LTTNG is also lacking in stuff of documentation right now. So, for instance, if you want to add the instrumentation in your application, what I would say is the man page of LTTNG-UST is the best place to look for now.
38:25
LTTNG-UST is the user space tracer. LTTNG-UST, I think. And also, you look at the examples in the code repository. This year, we want to create a developer documentation, a developer guide. But again, it's just a matter of time and resources just to do that.
38:44
Can you easily extract the stack at this level? When you use a space and you want to extract the whole stack at some point, is it easy to do in LTTNG? I just don't get... If you want to extract the stack at some given point.
39:02
Oh, no, no. Okay. No, no, no. Ah. Is it open? Okay. Can I go? Okay. Hi.
39:21
Can you describe, please, how does LTTNG... Yeah, I'm here. Sorry. Can you describe how LTTNG analyzes data while they're being created and what interfaces are available for doing that? So the question is, is there any interface to do the analysis of LTTNG data?
39:42
I'm sorry, I couldn't hear you. The question is, in the live mode of LTTNG, so extracting data while it's getting gathered, is there interfaces to do analysis, is the question? Sure, yeah, that's it. Alright, so we have LTTNG that does some analysis. There's this TMF, the Trace Monitoring Framework, I think it's in Eclipse.
40:04
So this tool that I'm going to present is a huge graphical tool to analyze the LTTNG trace and all the traces. It's in Eclipse. There's a rich client, what's it called?
40:22
RCP thing, which you can use outside of Eclipse, that allows you to analyze data and compare it. But if you go to automatic analyzes, for instance, no, there's nothing right now. And this is something we're looking for contributors to help us do automatic analyzes in terms of whatever you can think of, security, performance, and so on.
40:48
I was wondering if there's a way to use LTTNG in non-C codes, but without dropping through the FFI interface. Right, so we have Java bindings, and we're working on Python bindings also.
41:05
But, as of now, the Java works fine without using any C, but as of now you use C code or you use Java. We don't have anything else. But this year we're working for Python bindings. So it would be possible? Yeah, absolutely. Hi, I'm involved in a real-time Linux project
41:37
which works with RT pre-empt, Xenomai, as well as RTI.
41:42
So the last one is out of question because that's FTraceland. I don't expect any problems using that with RT pre-empt. The open question for me is Xenomai, are there any data points using that for trace using LTTNG user? For real-time? With Xenomai userland threads, yeah?
42:01
No, so the old LTTNG code had patches for RT pre-empt, but the 2.x doesn't support RT pre-empt at all. It won't work. I can turn the question differently. Are there, in the generation of a trace event, is there any system calls, or is that just memory to memory operations?
42:24
Yeah, there's no system calls when the trace point is being recorded. There's not. So we just... There's the write on disk, but there's kind of a zero-copy thing, where you just mem-map some region of the data, and then we extract it afterwards. So during the trace point, there's no system call at all.
42:42
Okay, thanks for the comment. Hey. There. Yeah. So how does LTTNG compare to SystemTap or DTracer? So is it that it always traces everything and you analyze it later, or what's the main differences? Alright, so the major difference is that DTrace and SystemTap,
43:02
when you trace user space, you go through a kernel every time. We don't do that in user space. We don't go through the kernel, so it means we don't have syscalls. While DTrace and SystemTap, you have to use the kernel module or go through the kernel to trace, to extract your data. DTrace, it's great,
43:21
but the big issue is that, and I'm sorry if there's Oracle people in room, but it's been bought by Oracle, right? It was a son. It's been bought by Oracle. And then Oracle right now is just not saying it right now, but it's trying to put it on a closed source way of doing stuff
43:41
where it just closes everything. And so there's a Linux support for DTrace, but it's not that good. Usually DTrace is for Solaris, and OS X is using it also. And SystemTap is a great thing. It's just a bit complicated. They do stuff differently in terms of buffering and extracting data.
44:00
But again, they go through the kernel. We don't. And this is a very, very different approach in terms of performance. So LCNG is performance centric in terms of the features where SystemTap and DTrace, they don't do that. I just want to mention that SystemTap and DTrace have dynamic tracing.
44:21
We don't. That dynamic tracing means you can just, your application running, and you can just decide to trace that specific function, for instance. And they go through a kernel to do that. That's the difference.
44:40
If I want to trace a user space application with UST, and I want to get a full trace from the moment my program starts running, so all the events from the beginning, can I somehow pre-enable the trace points before the application is registered with session demon?
45:01
So you can enable any events you want, and when the application starts, we're going to enable them before the main of the application starts. So this means that when the main starts of your application, the trace points are already enabled. So yeah, you can enable before. Okay, cool. And one other question.
45:21
If I have a long running application I'm tracing, and halfway through the trace, I decide I want to enable more events, can I do that without stopping and restarting the trace? You just go enable event, and it adds it. Yes, well that's the beauty with tracing, is that you can enable, disable events whenever you want. So at any point in time, you can disable events, meaning stop tracing that event,
45:41
or you can enable them. So you don't have to enable them prior, right? So let's say you enable one event, and then you start the application, it traces, and then you want to disable it and enable it. It's not important, you just enable event whenever you want. During one time.
46:09
Hi. Are you guys looking at tracing guests in a virtualized environment through the hypervisor? So are we looking at tracing virtualization?
46:22
I mean, you have a session team running on the host, and then you have a couple of guests, and then the guests are basically tracing. So KVM has some events in the kernel, so you can trace those events from the KVM events, but getting the trace out of the VM, within the host, for instance,
46:42
no, we don't do that, but we support to trace different, we add the host name in the trace. So for instance, if you use LXC or V servers, and also contextualization, in the trace, it's going to show which host name is being used. So this event, you can say, oh, this is coming from that machine.
47:00
Virtualization is much more split between the host and the guest, so this is much more difficult to just trace. So are there any plans of supporting that in the future? Not now, not right now, this is kind of difficult, but one thing would be great is to just synchronize the timestamp, and there's no plan for that, and that would be kind of awesome.
47:22
Thanks. All right, so thank you all, I'm here for next weekend, and I have a talk tomorrow, so please come to this OTR talk tomorrow.
47:43
Thank you for the talk, and the next talk here in Johnson will start.