A Comparison of ftrace and LTTng for Tracing Baremetal and Virtualized Workloads

Video in TIB AV-Portal: A Comparison of ftrace and LTTng for Tracing Baremetal and Virtualized Workloads

Formal Metadata

Title
A Comparison of ftrace and LTTng for Tracing Baremetal and Virtualized Workloads
Title of Series
Author
Contributors
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2021
Language
English

Content Metadata

Subject Area
Abstract
Tracing is awesome. Full stop. But what tracing? In fact, even just on Linux, there are quite a few tracing solutions, aren't there they? In this session we'll show off and compare ftrace and LTTng and, for visualizing the collected data, KernelShark and Trace Compass when tracing both baremetal and virtualized systems. With tracing, we can get to understand what is really happening while our software is running at a great level of detail. Depending on the adopted tracing solution, that applies to user-space components, to OS kernels or even to both. The collected data can be very helpful when investigating bugs and misbehaviors, but can also be used for understanding the internals of the system and for chasing performance degradations or improvements. On Linux, tracing can be done in a variety of ways. This talk will focus on two: ftrace and LTTng. The aim is describing their characteristics and show how each one works. After that we will show what are the main differences between them in terms of what they are capable of, how the are used and last but not least the overhead they introduce, measured with several benchmarks. We will also cover the tools that one can use to visualize the often huge amount of data that is usually collected during a tracing session. Both textual ones, such as trace-cmd and babeltrace2, and graphical ones, such as KernelShark and Trace-Compass. Last but not least, we will offer some insights on how to use these two mechanisms for tracing virtualized environments. I.e., how can you use them for tracing what is going on both on the host and inside one or more guests, at the same time, and how good are they for such purpose?
Torus Execution unit Software engineering Sheaf (mathematics) Student's t-test Student's t-test Tracing (software) Different (Kate Ryan album) Visualization (computer graphics) Software Universe (mathematics) Software testing Pairwise comparison
Game controller Functional (mathematics) Overhead (computing) Divisor INTEGRAL Graph (mathematics) Multiplication sign Disintegration File format Set (mathematics) Mathematical analysis Open set Tracing (software) Computer programming System software Power (physics) Exclusive or Latent heat Labour Party (Malta) Blog Different (Kate Ryan album) Kernel (computing) Software Spacetime Information Exception handling Source code Boss Corporation Overhead (computing) Electric generator Graph (mathematics) Information Computer program Open source Stack (abstract data type) Cartesian coordinate system System call Latent heat Kernel (computing) Event horizon Function (mathematics) Sanitary sewer Physical system Rhombus Spacetime
Axiom of choice Source code Perfect group Observational study Building Connectivity (graph theory) Multiplication sign Source code Heat transfer Tracing (software) System call Tracing (software) Component-based software engineering Network topology Kernel (computing) Telecommunication Operator (mathematics) Software framework Spacetime Software framework Modul <Datentyp> Office suite Object (grammar) Spacetime
Point (geometry) Wrapper (data mining) Multiplication sign Login Tracing (software) Computer programming Different (Kate Ryan album) Computer configuration Object (grammar) String (computer science) Single-precision floating-point format Spacetime Library (computing) Software developer Shared memory Limit (category theory) System call Thread (computing) Similarity (geometry) Uniform resource locator Computer configuration Function (mathematics) Network topology Internet service provider Object (grammar) Spacetime
Curve Group action Functional (mathematics) Perfect group Code Ferry Corsten Multiplication sign Physical law Binary code Compass (drafting) Shared memory Code Graph coloring System call Computer programming Connected space Wave packet Frequency Process (computing) Causality Personal digital assistant Computer configuration Operator (mathematics) PRINCE2
Point (geometry) Slide rule Implementation Dialect Run time (program lifecycle phase) Plotter Interface (computing) Web page Magneto-optical drive Theory Parsing Discrete element method Tracing (software) Graphical user interface Data management Computer configuration Revision control Implementation Physical system Resultant Descriptive statistics Task (computing)
Point (geometry) Dataflow Game controller Multiplication sign View (database) Plotter Compass (drafting) File format Sheaf (mathematics) Mereology Plot (narrative) Field (computer science) Fluid Blog Befehlsprozessor Smoothing Task (computing) View (database) Mereology Frame problem Process (computing) Personal digital assistant Software framework Table (information) Task (computing)
Asynchronous Transfer Mode Dataflow View (database) State of matter Control flow Group action Evolutionarily stable strategy Tablet computer Personal digital assistant Arrow of time Right angle Interrupt <Informatik> Physical system Task (computing) Task (computing)
Dataflow View (database) View (database) State of matter Sheaf (mathematics) Control flow Thread (computing) Plot (narrative) Event horizon Befehlsprozessor Royal Navy Process (computing) Physical system Task (computing) Task (computing)
System call Cone penetration test Latin square View (database) Multiplication sign Frequency Population density Read-only memory Semiconductor memory Kernel (computing) MiniDisc Physical system Form (programming) Task (computing) Dot product Graph (mathematics) Theory of relativity View (database) Statistics Population density Personal digital assistant Table (information) Physical system Task (computing) Reading (process)
Observational study Principal ideal domain Code Multiplication sign Water vapor Event horizon Hypercube Data model Virtual reality Different (Kate Ryan album) Befehlsprozessor Visualization (computer graphics) Vector space Integrated development environment Process (computing) Information Interrupt <Informatik> Task (computing) Physical system Information View (database) Sound effect Median Tracing (software) Event horizon Workload Integrated development environment Personal digital assistant Series (mathematics) Revision control Game theory
Intel Overhead (computing) Multiplication sign Stress (mechanics) Food energy Event horizon Number Benchmark Befehlsprozessor Set (mathematics) Computing platform Task (computing) Overhead (computing) Stress (mechanics) Core dump Tracing (software) Thread (computing) Process (computing) Workload Computer configuration Event horizon Computing platform Software framework Table (information) Task (computing) Resultant Row (database)
Axiom of choice Boss Corporation Overhead (computing) Morphing Overhead (computing) Kernel (computing) Multiplication sign Compass (drafting) Spacetime Tracing (software) Spacetime Power (physics)
Point (geometry) Functional (mathematics) Group action Observational study State of matter View (database) Direction (geometry) Archaeological field survey Streaming media Event horizon Computer programming Automatic differentiation Different (Kate Ryan album) Energy level Information Analytic continuation Category of being Identity management Fingerprint Physical system Beer stein Scaling (geometry) Information Software developer Element (mathematics) Demoscene Word Personal digital assistant Video game Speech synthesis Row (database)
Element (mathematics)
hello everyone in the stall korean begin to the difference between the dingy and after race i mean you're a university student based into your in i'm carrying out an internship at schools where i'm working with the audio for jolie software engineer.
and the tories nation spatial least you get the now even in the thirty's a shrine section i also want to thank my two daughters rico been a professor at my university.
today we are going to talk about race seen these practices are used by people to record information about a program cigs exclusion for the bargain proposes and that the schenone to the most common problems within the softer we are able to trace boss.
this system software and the applications the difference between tracing the and logging here are the racing usually much more detail that they became be in a boulder in these able to that's fine grain labour and eight the came be very noisy a usually. as more overhead the we are going to analyze true of the most use the tool in these for you. starting from an to injury also known as a linux traced to keep next generation we can say that it is a powerful open sewer set of tools to trace the linux cadman a user application of the same time. the distinctive feet or so are the colonel a usable space tracing a spice so follows the commonplace for money to buy every face for not designed to be very fast to write without compromising great flexibility low lot in sea and low over. read the tracing traces can be analyzed the offline or any other time on the other end we are best races are also called for internal tracer it is an illness care no feeder that lets you trace a linux kernel function on calls. essentially it lets you looking to the linux kernel in see what it's doing the turnaround few courses are integration in the linux kernel most people in the fare in the places for example from from graph a stock or are your. wake up except for more than one thousand even some specific tracers for lot and gennadi these the factor that the chase infrastructure we have other her no performance tools so for example better for control it. from specific feisty same are specific tools for now we can see how you can only starlet these two schools starting with the most simple won a tony starlet the teaser f. tracer that's because it is already integrated into.
the current and the if you want to use a user space component to you can use a place in the one thousand and four at indian gee you need to install the package is meeting the from the sources are using the niesr pocket g.'s of the objective of the stall could use to come. paid the two tools to do that through we are trying to trace perfect bench home on the it is a general framework for going to park street the commonly study year is creating to process season that executing one me on pipe operation at the office and thoughtful time. me that the executive for helen and play seemed we can see to the frontal transfer the first won four in the dingy we create a new say shown up for great singer enable the uk answer for shed their name and ford or read them right sees call used for pipe communicate. phones end the four f. tracer we are using trace him the that we want call money starts the precinct we've all the violence least the before it without a trace or to trace the user space are the best choices the iss the the ltd in gene the toll first.
difference me thought so too do it the provides three sephora increase log r.p.c. me out to preen effort or write your own race point provider but these parties out of the scope of these two or more interesting for hours that are the pre been to use a space a tray single. first listed in this light the they are shade object to reach out to article instruments east them functioning cosier to print all these share the object to your program you can use that as the pearl owed four after a cease the that we don't every adult that not be too ill to dingy because. the only option he is a trace market and it is very limited race marker provides a way to right into the tree string ba for from user space of these marker we then appearing the race to give you a location where a specific event awkward but are you can see. you need to have the call the two of these marker and others saying time it will cost you more development time whereas the heir to the n.g.a. offer besides these are put to needy pretty the user space for a single person as we said before one of the advantages of using the use of space tracing.
is to be able to understand what is going on while the programs running and the connection between the action exert use in the in the caramel side and those executive into users pain some of these example we use these code the main function and crates that end tread the with the the trident.
body function as the cause of each tread the after that that he will wait until then the world has read some the trade body function on the other end that prince are on the terror mean our history creation of the law could during these require into acquiring parodies in on the tray. case compost to look we can see that the process with the tea the one for four nine has required that the law then not if called the food texas east call lot probably because the locker was already taken after a period of time when the process with the tea the. the one four for eight hours finishing the operation with glaucoma you cause the foot texas color to wake up but also the political process season after their the israelis the law curve when these leisure and the process we've the the one for four nights a week. you wake up the exit the food dexter ceased color and then i acquire their not could if available talking about the three sizes are for the perfect bench share price common the we can see that their city in jeter a search is very small compared to the. if the race race and probably because a dingy uses binary fi's whereas the f. train he said offers law that you know you will redouble forma after the raising his and the we need to analyze the lover for these occasion we have true option the first one is to use a.
the interface and therefore bubble place for and fifteen g. its reference sparser implementation for city effort in the very same the for a race are so if not needed because the race can be read the also we've caught common the for example but these more convenient the. second option is to do a graphic can are these are the two e.c. later in the slides i don't have much to say about these two tools or because they are turn a very senior result for each or are you will see even report to weave the tay description if you want are easy to use tool. i will choose bubble trace a because the dial are more you worried about where i was for the same the you need to know all the roads through poor to understand the results for example for the sheds reach are lying these and two points in the manager are well. these he bought in the bubble trace of these one indies will compare to these one in these one now we are going to see the graphics tools to turn a shark a that offers perceive york t.v. to plot the emperor task of activity plot.
the star mark said that they want to show our are the provided the user are very clear and the use phone is seen at the the real meant that there are locked in these are even we even not so big crazy season in these imagine you can see true possible marker. the. show the also in the plot section. and there in the table section then that between these two points out their mobile phones that used to move in zoom in and out the plot to in the bottom section there is a table we've even answer. other of able feet or are the field painting pursue few task or even some east think the trace comb process is a very subtle tracy's a long season allies are that supports multiple tracing four months in our case and indeed the ball for the tracing tools. trace our support to a big worry to abuse a summer we church we will see after these lived before that it is some remarks a trace composite is part of the eclipse frame or could there are no locks are you access stays most. in the fluid talking about views are the first won and we will analyze the in the flow control of you eat his usual because a it shows or what the task is doing during the press i speedy of the time on the left that we can.
see the possible say that that the task can assume each are represented ask are on the left that there is the name on the right this autumn's in this case that there is no she's view the journey is not made the base don't ask about pays the on c.p.u..
aso year the legend represent the south have seen which the past eason this if you plot the is dividing in the finance section the first corner all show the task running the second want to represent the stops and other are far i urge you. even so these views an hour for the colonel three senior but we have are also other possible views like the c.p.u. user drew that doesn't the name saying he said it chose the even we've got off so how much c b u u s.
is a used to go each task this system called la things you you that is the abiding others have views are shows duration and lot is he all beaches system called these one represent the density be is a represents the last things in the form of table. these are the lot in see such the stakes are these one in say they represent the graph latin superstar i'm in this case a the blue dots are are blurry the system cone and the red won or are for the right. system phone that the sky your view these one shows the reading and writing speed the for the defendant these come this east them and the last one the memory use a view on the right side the of the view lost the relative amount of memory that was a locating the end they are located. doing these pieces that period of time.
so everyone's the study fossils sell it tries to environment that we ever lost and that one of our guests and typically the waters of interest which is the one that you want to trace the us inside a guest also a if we are using a.v.m. other guests visitors abuse are visible in the last hours regularly starts.
so what can we do. well we can trace the all-star we either streets or entity and g. and we would see the events pertaining to the adviser for he starts in a game everyone yelling creature we will also seen the tasks that corresponds to the sheep use of the m six when the and all the events related to them for example yet we see that to be next. using guest code the heene corresponds to the cave human preventing greenock and actually stop doing that that when we actually give us the new but to really have no way of knowing what happened in between in setting itself. or we can trace the guest and that to be seen as to treating other metal system in fact we will see the events are related to the work of running the guest about not that was used in effect the data they have on the austin on the adviser for the stance to steal in the same traits got to go under or so we can see she added that you to spew wants to write them. all of the creationist these must mean there is art on biggest killer and try to do something but just with his information show we can not usually tell what. so it would be good to be able to trace boast the last and aghast at the same time and then combine and synchronize disease we know that both of threats and the median g f some ways of letting one do something like that in fact we plan to investigate and one but is only their mind that of amusing to she put yet says. one of the difference to him green to choose one two or is there were a in our case that we trace the following two commons.
we know tracing with a tracer with the engine running them or two hundred times and each the platform use that can be seen a year after that bench exact who shown we can see that the pond. the first row of the table there are over ed also you only this kid running un enabled on the second the role the number of events enabled are all a for bafta tracing tools are and they're already israel year i have the same time you can see that. celtic dingy overhead that seems a lower but you need to see their that after a search is probably tracing more remains the first thing that i want to point the in these a stress energy taster is that the negative values or can be due to the experimental. terror after that that we can see that the overhead there is more like almost the same for both tracing tools for in these paintings are the overrate bodies be the number of task use the i almost the same for bafta racing tools and despite their that number of the brain. it's being traced that there were either the small probably be khoza after the process use around noon they are operational run in their users pays a in conclusion we can have the following comments on the previews results for can and tracing after a single tinge.
you are a quality powerful news for but that the same time after a such as the big advantage of being integrated thing the colonel morphine disney the to start using it as a team g. is the best choice for use the space race even for the graphic to. also can a shark is varying the other three same the from ten in seed the race called boss is kind of the comes to sleep sir but much more powerful and the flexible the tracing overhead the his boss for work wrote that a lot dependent and. the red the introduced by if tracer and the dingy is pretty much the same thank you very much for your time in the future of quesnel we would be epic costs for them.
she myself if it doesn't really also or let's try. i each year question above any comment about street can i answer. can i answer he still has that than typing i guess the real try to answer life he started typing and the that someone stops the doesn't work. the study said he uses an ice but of course a lead see you. the survey and the north up and serve. as far as a system called cider concerts all when they are called up from just a stroke to hear what was a nice for example over entity indeed it to you found out to and what it is also growing threat of other solutions such as a. using new probe or i did we see them to or even day to be the greatest feat was just in which they honestly you know it to be were also and i want to try ease the day you can achieve much finer greener grain cereal. control and much trying again. china already that's what they wanted to see. the. function might be called sirtuins did he said most explore swann we treat mention in the year which is shown in the example of what actually up and sit inside the care they may learn not only from the point of view over we choose to call the book but also said helen he said dance to. to the fact that you can respond. so. in their functions of the killer not only use the difference what you want to do your we usually just mean something else. yeah i and it's also the fact that it relies on peter razor and the sole the overhead and everything. yes i think eight so that is true of me you should for as far as the with bt year for which i think so i sold or was it the blast. open to some in the aisles to bolster probably but i'm not sure from according to what i think it was about both states and also the same top are reaching to wall yet for the poor soul. there is debt said before did indeed remained reason why the deed in the main journey into a cultivated saying easing is what is that we are looking get something different. perhaps while the people think about what questions if that any i was just a couple of words more about the transition partly because i very quickly said during the talkies going to be day they reminded a continuation over. years into sheep. we've been able or a the to use a both seen another work and also getting there. whether we really have been any air be able to use a. f.t. seen the u.. we basically with the same the. a fact it does an agent functionality that you can start inside of yemen we're able to use this recession you for raising both costing yesterday and that they use also. both increase in the end careless chart. development direction so you can only to sleep. but already have some a facilities to synchronize a row. the year. i had a good understanding all. both as a guest and you combine we conserve so acute a sea of the synchronization about. looking nice for us to change the entries compass serve we know for example you just from the british n n from under the hood that should be possible to train one as any. so just sort of by so whether or not it's really forced to and the way it works or. yes and the ease with added. the way. and the nominee question on with the state is i you'll die in a weakened could or could actually uses. the shoes when countries be too much information from users based the body well. she is advanced what you what you're doing what you need to eat a single. very a good the way the offer. winning what arts scene your sister and i mean they weigh your being away your first seeing what up. and from one random function in your views of speech program the study of the school. the and t. want it on the functioning side did said occasionally the or maybe not so much or and the arches much that you really care. i think it's a useful for example to one that stand the war. but some appreciation what you see some doubts. the. i told evidence. in denying the identity she has been useful for development and the bargain both scale and. occasionally shoes or but yes of course if you you unable to many. the race points or events or whatever and you end up with any be cases that was so they can also backfire and. could be very difficult to analyze everything yes. they need to be here for the ira bunch of creating solutions are all over the church. using gets enough to make any business. been you can enable out of events are two levels so it's easy to get lost or you do we want to use or into the stream or events that you get. powerful i. i think it's good to ever lots of stuff it's because a lot of stuff rather than to ever fewer of them or even use these means that the us we can choose and the be careful what you enable. it.
Feedback