Linux tracing with LTTng

or to find it parsimony is there there you worked on a deal to the new

projects are also on our data shows which is a company that basically we make features in the development of thingy sold today on the news so usually tracing thoughts or can cover of the boring but I try to do my best to find the so just show around here who ever heard about delta g g our experience at least 2 people with and so tracing and in general our individual events uses raising a would Castries her 50 and as results of our right to so I'm in good track uh so I

maintain the altitude you tools about part component of the of the G to change and this presentation is about like tracing builds GNG that they're showing you what we did in our in the last 3 years in terms of excellent feature in great work and how you can use it in user space also and so there's really a quick overview of the change trees in general I had a problem people here are middle know what procedures on just going to be an overview of all trees and all of them are going to talk about originals and some future work and so the difference is interesting

and avoiding or and what log it's a slow and so on that is that you can enable or disable advance during runtime and tracing is low intrusiveness very high performance so we delta g in g of we created this user server and some federal Teresa and introduce entity tools component that makes them together so we can merge Carol endless pastries with and what within 1 trees and analyze that after all so tracing is basically a high throughput by performance debugging have like that for you print out and then sold 3 uses that print of but I performance written it's killed CPU's also so this is this what's in it so for instance has 3 so anyone knows histories you agree to the users traits uh and just to and understand how to pronounce the DSS skulls but this is also interesting is more

is different from from the bottom it's very

important to understand this this is the basic truths about the spoken about to make use of 3 senior the tool of that Institute of X so we have to look for it and bring along some of is the unified tracing so if I continue to use the space and kernels but as I said were lower here we use a commentary for medical here which is the tracing formats in binary format on disk and then we analyze them with a few words what whatever you if you were to use and we're now all shift in the wind into the readout in other distributions so as I said so there's 2

traces where you entered into modules which are in the kernel trick so you as a and the repeated that like a lot you don't need to recompile external until the DAG use modules was now so the difference between a contingent that between person CNG our is that person's internal they do a lot of the stuff they also on the profiling where else did you doesn't do that but with the kernel model with the kernel 3 so you can trace lot of events in the kernel skulls and a bunch of of of whatever happens in a kernel in the different subsystems and then we created this you receive the year this is 1 most eminent part about today's these user this research so the love development will print so the news tracing user space and so

this reasoning process libraries so you gotta you instrument application recompile and you have the trees but the good thing is that enable these able and during runtime any events begin life traces against snapshot there's a lot of interest in for but yes so you can use the

kernel as into 2 . 68 before at least to digest the kernel but after that no on the modules are the utilities

of of the TNG so at the end user of our created a out of demon scholar session even the the session the first 1 is the freezing registry daemon this thing will combine everything you do so when you create a trace for instance so you have and injured create would come line so you can use the kernel in the spaces so it's pretty pretty great where you can integrate our whole slide into a tree of so yes so there would predict control library can use as else C so it's a C library interlibrary rather really which this really makes but you can stream the trace of a network so we can extract traces from

machines and put it on the network and then analyzed later on so this is the really needed in streaming data and the consumer he is the 1 we extract so there's a bunch of even is you don't really need to care about that when you trace will hinges so uh but if you want to do that contribute to or make development entertaining those kind the component but you were as

well I with choose which basically takes an input phrase and put dietary of traces and file traces and just print out on the on the council of human readable but there's GNG thought and this is

pretty pretty often so everyone years no stop the each stuff and this LP-CNG top is exactly the same but it uses kernel traces to print out the information from the top so the difference between something he does that we don't truly trough every 2nd for instance we just use life traces and we you can and you out what going on this pretty pretty often it is usually performance or of 1 to 3 per cent of 5 per cent it on your system worked out it takes a lot of use

on art so this is altitude user-space I you answered your application and I have some ideas about slides where you chose what's the what's instrumentation what's means that would call that tracepoints so just to betray he application and then extract at high your your data in Brussels library and a session you run your collected traces and then analyzed that's it that's all and so on so this is a lot

quick excuses scheme of what's going on so you're in machine any of your session even position even as I said is trees and register it just helps you create sessions and enable events that so on a command line you at the end you create enabling than that she remains that uses space for dash K dash kernel and we have a lot of people that domains rapid other domains that I'm gonna show you later on and actually also you able all around orientation so we don't change a great deal about that this issue of instruments addition of you list what's what which application you can trace so

if you start her application on your machine and didn't register for that session Newman and solution even with by consumers

safeguard Camden consume your trace an extractor trees and that's it's pretty easy an incident the attitude and the answer to that station will have so far consumer we create our profits and we share in the application so that it is you just pop enables profit to trees in the data and this is the great thing about treating you space with use that when the differences when you application crash as well the the buffers loss of life so we i'm offers we have everything to consume them we notification is that so we just extract you know and then you get all your trees uh even though addition

that and this this is done with a shared memory so we use is we are using every uses the source you so on and 1 of the 3rd or Syria assurance or see you yeah sources of computing but it's so users resources that progeny means really

company update its and synchronization mechanism to have very very high efficiency and performance than the right performance of that structure what is the the structuring in as sewing users there's there's a library of course you library call uses course you can check it out and we

use that a lot just to synchronize traces how form speeches as I said if you have 80 use 1 to 1 28 to be used on a scale of over and so that's what sort of so yes this is basically all we do stuff

in userspace shared-memory we exchange offers minutes our right to start and

who so this is this value in certain applications and the

status of this see it's macros it's kind of complicated to be honest in our area of how various elements proof that helps to period of a year and a half years Brussels the but it again you will be the the usefulness person is that you instrument application you do once you don't the print as everywhere who just instrument put on trees points and then you can trace it over time so your your petition is running then at some point is in problem well you can just be able call events on the tank is the subsystem is my eyes guys were in this value working and you need to make such a data so this that is so this is a test adhering to the entity and Europe history we have a and B have 2 examples and this 1 is a moral example so you provide an interest point main those are 2 different used to separate trees points so you can have a look at a bunch of tricks point unification interests that spring and to providers or domain or whatever so instance of you in the company name and then the department in and the application a subsystem and that's it you have to provide a name and you can just raise the fields that they don't want so you recorded in stream structure whatever so soul whether the basic function here I'm and then that industries point that the unit of created here so you with the test all and then we using it right so when you're using only on when you're coding application or maintaining at whatever the gain of 1st point to the cold and this series on put trees point and then the fields and is very record so this interesting is not enabled and there is absolutely no a performance is so the twist of the digits of zeroed out and that's if you enable it during runtime and then you get but I did area in benchmark the performance of child's but as of now we are around 250 nanoseconds it takes 3 application to the trees point and 2nd the lowest annual right now there's no sense going there in the back it's all of us you will copy stuff so this is very very i performances pretty pretty pretty good stuff and then just going person session so the command line you create a session so that the 1 enable the subsystem number 1 subsystem the number 42 and can use wildcards like that's in just subsystem 42 underscore in every event and that's a system would enable so that you use a space and and the sub system name and you can start and the other thing you start means just spike this the the tree is going to get recorded and then you can see the way in whatever you want coffee and then stop at some point and you can do it so that this URIs so this

is an example of of the trees can extract so the other times than the the top of the but that and then we have the delta between those 2 events that are all ranged in is the delta between 2 events so once again between this little since then they also in their true interest point name provider interest within name and then build a security saying this event was done on that the use of you want you to speak to and then the bill and others so for the building can put whatever you want to know anything strings in its that this structure pointers again extracted as a which is a human readable output that means the from babble trees for trees and so this is this is another example of

the kernel so for instance this is goal region this is golden and the escapes which states which is when to begin pj sort them to private processes that gets which must get during the kernel so so yes for instance we have the bill really the bottom point an account opened the affine flags sums of our itself up around the same outputs and in terms of of culture and that is pretty similar interests also because of histories can understand you the layout of this skull for instance if you do not receive messages send message and appeal to the uh you know the type of of the pills which means the structure of the of the arguments and so on so this is why offers appointed we don't know if what weight points or what's the structure and this just point but again human readable

so again spots that link and you

but this is not very useful India and because he just started trees the trees and then you start

so dreaded snapshots was not just means you're running your addition flight recorder it it's so we buffers you get sold you trace data you get a data but in the offers at some point in time uh depending on the size of the buffer is going to get rid of the so it's going to get up and you raised by new new data so right we modified recorded the point of that is that let's say you monitors your your answering application we don't you receive Twitter is the the width of interest points and then you start he just be able everything started just forget about it is goes into flight frequent mode going bottom-up and you have some tactile you an idea was a lot and the true

snapshot so what this does is that it requires snapshot did everything and from that point on the destruction is then extracted you wouldn't this or part of a network as you choose can predominant working ascended to a redhead now injury ladies in storage and file on another machine or on local commission and you can then analyze the traits that getting rid of digital why users in is that you can do that with a kernel and user space so you can snapshot boat domain both races so any because a timestamp or a synchronized so you have the trace they are from application and from your kernel so this is pretty great produced in you that from 1 to monitor

application that runs in production France

so yeah I created enable spot anyway you something and you statute records and that's so real world use cases where we we use that and it's it's it's then it's the main news flat where we go on to launch a large set of the news with that is for instance we are the core inlet accord up the way of course dominant implementation cycles whatever anything your application called dumps well you can tell it to you when it caught snapshot and is really really useful with especially kernel begins again knows exactly what happens which applications and the kernel when

this obligation caught up in as I said the buffers are shared with the consumer it means that when he scored on object you have the assurance that you get a get every event of the triggered by the application so we have that tree Ronald initiatives to

entity genes that have I did not tell the lord of the ripples through there and you have this we have this fancy you bash script that allows you to just a snapshot when accordance the idea also so let's say you are tracing regularly on your you just raising your your application you kernels you on different machines for notes and there's an there's right now research down and University in Montreal where it uses the data on data that is uploaded from L to the G to treat AIDS to analyze that on the spot real-time and just triggers trigger action made some some pattern emerged and sold at the being used to correlate data

logs in to merge with syslog also we have that we would try to work deployments that about allows you to that use tracing the and put it with the for instance plunk for or like on in terms of time duration uh you can use it for the performance profiling and so on and so forth so the hassles last year the since september about media we are assuming the recently a process for the 2 . 4 reversion but we had the light components slow-light component means you

can trace your data you can trace prone and you can read the event as they've been caught that sounds that seems like easy to you but it's not very difficult so the interval synchronization performance to achieve better performance on this but I have to a lot of work in the media we added this feature upstream so as not only in for instance uses that you traitors to trace the recession you trace your Kronecker education and you can lie read the traits so 1 of the things we're trying to do is to have a the and like in a drop in replacement for as for as for instance where we can use of CNG to do the same thing as history the in the different answers now the is that a trace changes reputation it uses betrays its changes staff and

did you hear a petition wooden x-rays and without a series of different ah because the truth is very intrusive intuitions registry in new doing some uh John trampoline offers an to to uh to extract is the sole imagine you can use a girl with the 3 forces to persons that its performance on your system and the exact same thing with this sort this word trying to this is why we come here to files and just like I explained in this and that's a feature not sell it to you but also trying to make you contribute and help boss to do development of contributors and changing so this is a kind of work so the

light using the thing is that you can countries analyzes clusterable analyzes you can do a lot of things which life imagine where you can like the life and 1 of the thing we just change in the last years that we tried to bring tracing to not kernel the block hours right so to use a space of opposite system administrator just to use tracing in nothing development fashion between just monitoring useful way so as a quick fix about image

of of the infrastructure that is usually of of the station the 1 year olds so this means else session use the demon that how traces of the trace of being collected on the other thing is socially and thought arrays the of the those running traits and you can extract on the to really be an adaptive you were connected to that region over TCP cities are so for instance River I we work with the companies uh large companies like Ericsson in Sweden that you use kind of infrastructure where they have you demand of the extracted data to a couple of moderation true network and indeed you which would or a machine to analyze it through another through that network uh it

scales on a massive amount of of of machines and it's being used so it is not just like you know

just 1 that's of someone of some developers having a dream it's been used so yes I only I want to add the more you have to digital but

a I got a problem with my out this morning so I can't of unfortunately but what I can do is use just to

the just to show you some some basic evidence that so what we have is health thing here so LC-NMR gene and this is something that will be for this so 1 thing is that if you have a session even running as roots i is it you know that Natalie kernel module kernel trees if your application is your guide the user is in the prison

possibly registration group and you can you can change any means is comparable with a the freezing group allows you to talk to decision even in the session is running as a random trees kernel so normal user trace a kernel soul mind that recent groups some of the things you

create this session in your soul there is no such agreement so

strong and going around the trees their time so healthy and you in this last case that means use the event to the kernel domain yes signed the GUI and that's an interesting part of

me put you on with groups

sorry but my like is a mess I the the Robert put on the back of the room

stuff of help you is that is so

this is all the kernel invented interest see if your 1st 60 Trudy are accused the of loss standard advanced of our of our internal right now it's tracepoint article traced back to the principle of how the same events factories also at the differences in a single vessels and then you

can trace whatever you want so let's say we have going to let's 3 the

mole and see are much events in create some just enable all them to my session for the current look into session as I could and should stay and it is so all

the enzymes and there's a lot right so when I do something he starts so I that at this point

my mind a computer is atrazine Detroit just to show you how much they're going to create and changes in this search so let me just trying to

center so we can see 11 11

megabytes and then 12 megabytes or a

nice so those are the the Lord the

your deal on the file after that have been filed so channel 0 mean just shows the was of the 4 channels synchronous channel the I did put on events and involves China so you can have different events in different channels and different session you can move that around in the network and so on so as the nice tracing an itemized viewers about doing nothing but i'm extracting every every event the and so a lot of the and there's nothing this store and also that

so let's stop there and the show you how it looks like yes only if the don't so

is a bit of its nonstop I

have learned a lesson to you all for what's months by so there's not being recorded and this is true this is kernels that every 10 Ramallah illocutionary oligomer attention every skills which everything in the argument is being trees so all that's 1st that and and just to show you the the

amount of data you can extract without any performance and this is a really great and you can as I said you can couple that without any you use user specification of sold if I have any application yielded change this that's you and you can see the application and what you can trace when application starts only application just sufficient even and and this is the event so you can just list that you see what efficient interests and trees a good that's all it's

still that to the the right so it

is something just performance results just to show you the the awesomeness of so we're on that for 15

minutes of our own and we took a snapshot of and every I don't evidence that you and created a snapshot of 7 megabytes of data on and you'll as to raise of so we compared to restore resources would create what about 5 out of 5 . 4 gigabytes of data in a matter of 15 minutes with 61 million about so remember phrase is this calls for a specific application entity you will trace every Cisco happens in your you in your machine for every application and the only spouses goals look at all the other events you so may create 6 . 8 to 256 storing to millions events of the same type syllables or and and we have a 1 cent didn't lost that so we themselves basically at this way too much the I speed and so we have to of news events unfortunately need to be able to have this kind of performance and record

so this is the the degree this for so mean it means that we have this and the trees in the input on that it's a dedicated to share this with your operating system and you can see all in their supply lines when he sees you right now sold the red 1 is this race with efforts my spilled my so-called and the rest is the same with changes so there's no performance so is the same as no freezing and we are comparing that with flight recorder as I said the we just recording recording point office and the rest of the yellow 1 ancestry so we expected the history on the network so you extract inter-machine and there's absolutely no performance it but you trace 250 million events yeah so when a shared this our it goes a bit more than he gives them the very i contention on the disk and ideals or off the roof on solo tracing of course high-performance and then you get your restaurants which is the yellow 1 and L 2 the use scalable so this is why you can see the the flat line of history says that when you go to 3 trends on a city on a 32 CPU machine invented but doesn't it doesn't got some point stop scaling and it just doesn't go into the where L 2 fugitives and the difference is that we have such a person you buffers and we tried very origin just of haunt cashline bouncing in a of performance problem you have when you have the training a petition so all of the 39th much of

the recent work in a futures so

some parts of this image is about 2 . x has been created in 2 thousand and 10 genes and each version is the name of the year of our in Quebec and Montreal correct and Canada so we have microbreweries and greater that we got very good results every we following the the alphabet so it's like 80 whatever we on and so too does the woods and they all 2 . 1 is Boston have stood up to is a of could of which is a very nice IQ from Obama we Montreal and everything sold 2 . 4 is like a package the 1 which is a remarkable now called please don't collect and 2 point 5 I don't remember what's the names in the outskirts biclique F so in excellent of a 2 . 4 we added the light freezing LSS we have also this snapshot of the 2 . 3 but as you can you can use them and 2 . 4 1 2 printer whatever and we had also you just or you still support so I only if there

is a lot shot developer here I'm not inferior jealous lover of me like Jell-O of there's a lot of people using Joan so how we have this sort of contract to a supportive and not supported added feature of the of using job-related utilizing so you eligible user along his library where you can you log repetition go so we booked we're really understand you agent in general and

this is books itself on on the job will you to library and extracted data from that and put on the trees so it's pretty nice because should arrange our can do not interpretation of 1 line like just chat with CG does to create your class so don't remember exactly but just to create a class and then you start the job of the of thing and you can use it to change of ideas with the Council and trace your job so ah yes so this is a performance gets but against the use

of our future work and almost and so there's a lot of fun question a of future work is that we have this year and we are already where tracing would be no talking along with the harm people in their people and we gonna try to put on the Ottawa traces interested so

once we can do that it means that you can combine traces kernel traces and use bases in the same in the same room arteries and Korea times them and analyze that which is this to be honest pretty awesome and it's you can do that right now on this use of pride SELF proprietary software from market of for is trying to full source the trees uh but by it's coming up and interested triggers and draw the courts of Portland and an automatic analyzes dynamic also instrumentation with d-trees doesn't and we don't should but we're trying so if you're looking for for Trajan's Forum open-source project to contribute or you know even a job or dynamic guidance mutations are important because they're trying do

1 yes so you project and that's nowhere Europe so in this question that's right yes I think there's a

microphone around our just there's 1 person with yeah further work but more so for that and just 1 thing for the question answered please sign clearly if you have questions so we can come around to your place to give him microphone and if you want to leave please leave cry of the others have a chance to get asked that question about so you mention that there standards of tracing lines and then install so that I don't have to recompile and his father couldn't stand you log in you can alter the same that machine for example on the disk by that you can look at this the I can trace what's in the so what I enables the everything in the car and then I want to log on this data and on the local disk that heats hands like in parallel so they're do I get rules of log of the entries in my trace data only so so you're saying that you we extracting do we get our goals events in the data the all of the dual working so yes there is a kind of feedback to wind in their own where you use yielded GNG events so you were extracted data we using of course the skulls and you see that in the trees so the more you train the more you created by far you can fill that our we have filters and USC and then is that come with kernel just removal to the duty was question and I and our question about time we don't the uh tracepoints dynamically interval running kernel and we want to do the this fact so apart so the question is raised by dynamically add to it points to occur and so our the trace events in the kernel or status so it means the code of the upstream right now and you using so what's existing is the kernel is called the profile for concentration mistresses so you can talk to any symbol in the world that and LGG supports so we didn't do it for every event again just have let's say I 1 trees that that function the kernel so you can do out with the you would have enabled and that's just function that educated so those or of we can assume that exist the current and trees and the simple you know where usually addressed coordinate by the beginning of hi I have some questions phone and you space place that some of these in the condition anywhere to like 4 people wanted from nearest canon programming to to use space probing somewhere because when I was looking for something when teaching Judean find much information the assailant what's interesting points followed by some interpreter and use it anywhere reckon so our our documentation is always a problem with the source of projects and and that he is also lacking in the documentation and also for instance if you want to add the instrumentation your application are what I would say so you can have the mandate of the USC is the best place to look for Councilman Joe everything US the US is used is that you get used to it and that and also you look at the examples in the core the this you wanna create a developer of the condition that the develop guide cry again it's just a matter of time and resources just do that can teII extract like consists of a decision of but when you use space in and extract would stack at some point is easy to do in in the in the congenial that I just thought it was a really extra that the road to strike at some given point all well now look at the at the time I the result of this OK of I have high can you describe lives how does and at the end see yeah and here I they describe how it in the analyzes they why they're being creative and what the interfaces are available for the that asked questions that is there is an interface to do the analyses of everything you I'm sorry I couldn't use is the question is and a lot more about the changes to extracted data 1 it's it's getting out by what is their interfaces to do otherwise and so the question so you have parts so how we understand that there's some analyzes there's this team at the phrase monitoring foreign word contingency plans so this tool that is presented as you wish graphical to to and analyze the of the and all the trees are it's our it's an eclipse there's there's a rich client what's called our city thing which you can use also the clips that allows you to analyze data and think about it but if you see our automatic analyzes for instance no there's nothing right now and this is something we looking for computers to help us do the thematic analyzes in terms of what we can think of security of performance at after I was wondering if there is a need to the use of the energy in known as the tools that yes without choking studio phi interface right so we have a job bindings and we're working on part of findings also have got desert now without the job works fine without using even now he used to go to or you just we don't have anything else but there's this you're working apart combined this so this would be possible to get yet at the the yes I haven't both in a real-time image project which works with RT and stand my as well as on so the last 1 is out of question because that's after it's like I I don't expect any problems using that with our Dupree and your question from these analyzes are done in the data points using that and for trees using a DAG uses for real time with the cinema use all inferences are and also the old and the new Kodak patterns for different about the 2 x doesn't support article on the contrary question differently argue it in the head in the generation of a trace event is the analysis of course or is this just memory to tomorrow the year there is no system calls stuff when the trace of the court is not so we just so there's a right context but this kind of copy thing where regions and maximum region of the data and then we extract that so there is a tracepoint instances of the things comes and theory that so all the stuff that LTTE indeed compared to other systems have already traces of is that it always traces everything and you analyze it later or what's the main differences are at so the major difference is that the trace insisting that when you try to go through kernel every time out we don't do that user space we don't want to kernel so it means we don't have to solve why do traces system that you have to use the kernel module oracles to a kernel to trace the tree structured data the trace our it's it's great but the big issue is that I'm sorry there's oracle people in room but ah it's it's been Bob Oracle right is there was a song about like Oracle and an oracle right now it's just not saying it right but is trying to put it on the central source so to more of doing stuff for integers closest everything and so there's a Linux support for but it's not that you are using traces for Solaris and and I was using and also and system that is a great thing they just a bit complicated the use of different kinds of from offering and extract data but again the use of kernel we don't and this is a very very a different approaches performing and the performance of the thingies performance as century in terms of the future is worse than that in the trees the and just want to mention that that the use of dynamic tension of trees is we don't know that I'm interested means you can just see application running you can just decide to trace 2 of 2 trees that have specific function for instance I'm to promote that's different and that and defined I don't try to use classification we used see I and I want to get the full trees from the moment my program starts running so we had to begin I cannot be somehow pre enable the tracepoints before the application is registered with session you will have so you can enable you can enable any advance you want and when the application starts to get a table at the Four domain of applications part so this means that the way to get the main part of reputation power features what I already enable isolated can enable like before OK 2 and 1 of the question and if I have a long-running running occasional tragedy and a halfway to the trace of the sort of want to enable more advanced can I do that without stopping and restarting the trace the features can enable event and it hasn't yet so that's the the which reason is that you can enable disable events so everyone has sold any point in time you can disable events like meaning stop stop for a single event or you can enable them so you got to enable them prior right so it a when a 1 1 event and then you start addition traces and what it is they to build our designated it's on mortgages enable them when we want the entire at the we find that the more you guys looking at tracing guests in which less environment through the hypervisor so so are we looking at at tracing of the visualization just as I mean you know concession been running on the host and then you have comical yes guests and because of the the trees so few United some intense internal power so you can trace those events from the events but some getting to trace out the whole of the energy and how within the with the old for instance I know we don't do it at all that we support to trace different about me and also in the trees so for instance if you galaxies for this services that also the revision in the trace initial which holds the EU's going through this event again of say all this coming from that which digitization is a much more between yields and the gas is more difficult to just straight along the the plans supporting him and not all of the analysis kind difficult not wanting to be greatest in of desynchronizing thanks to the north there's no plan have planned for that and I would like also to thank the I so thank you all area of foreign next week and I have talked models so please come to this What's your talk tomorrow and thank you for the support and the next begins some