Linux kernel debugging for sysadmins

Video in TIB AV-Portal: Linux kernel debugging for sysadmins

Formal Metadata

Linux kernel debugging for sysadmins
Title of Series
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
A deeper understanding of linux kernel would help sysadmins to debug issues, tune systems and provide better root cause analysis. This talk would provide insights on investigative methods and tools, that can be used by Linux Admins to understand a bit more about the systems they manage.
Keywords System Administration

Related Material

Video is cited by the following resource
Point (geometry) Existence Kernel (computing) Computer animation Software View (database) System administrator Multiplication sign Moment (mathematics) Integer Computer programming Physical system
Presentation of a group Building Game controller Multiplication sign Insertion loss Shape (magazine) Student's t-test Mereology Semantics (computer science) Revision control Frequency Mechanism design Semiconductor memory Ontology Network socket Physical system Focus (optics) Standard deviation Touchscreen Key (cryptography) Information Cellular automaton Keyboard shortcut Mathematical analysis Multilateration System call Arithmetic mean Kernel (computing) Process (computing) Computer animation Integrated development environment Personal digital assistant Order (biology) Musical ensemble Service-oriented architecture Directed graph
Greatest element Process (computing) Computer animation Information Semiconductor memory State of matter Mereology Coprocessor 2 (number)
Dot product Process (computing) Computer animation Information Forcing (mathematics) Core dump Counting Right angle Funktionalanalysis
Computer animation Electronic mailing list Instance (computer science) Funktionalanalysis Parameter (computer programming)
Medical imaging Kernel (computing) Computer animation Information Physical law Quicksort Metric system Physical system
Standard deviation Computer animation Semiconductor memory Interface (computing) Multiplication sign Online help Cartesian coordinate system Physical system
Demon State of matter Multiplication sign Source code Insertion loss Water vapor Parameter (computer programming) Mereology Usability Mathematics Different (Kate Ryan album) Semiconductor memory Core dump Resource allocation Physical system Exception handling Boss Corporation Touchscreen Mapping Block (periodic table) Moment (mathematics) PC Card Keyboard shortcut Electronic mailing list Principal ideal domain Bit Funktionalanalysis Arithmetic mean Message passing Digital photography Process (computing) Befehlsprozessor Angle Order (biology) Phase transition output Pattern language Right angle Quicksort Freeware Resultant Spacetime Web page Point (geometry) Ocean current Flock (web browser) Backup Computer file Link (knot theory) Ripping Image resolution Virtual machine Field (computer science) 2 (number) Number Latent heat Well-formed formula Term (mathematics) Utility software Task (computing) Condition number Form (programming) Module (mathematics) Code division multiple access Standard deviation Information Demo (music) Coalition Forcing (mathematics) Physical law Expert system Bound state Basis <Mathematik> Limit (category theory) System call Particle system Uniform resource locator Kernel (computing) Pointer (computer programming) Computer animation Nonlinear system Personal digital assistant Logic Network topology Video game Object (grammar) Musical ensemble Local ring
Computer animation Wrapper (data mining) Core dump Symbol table Physical system
Computer animation Information Multiplication sign Canonical ensemble Physical system
Process (computing) Computer animation Information State of matter Different (Kate Ryan album) Semiconductor memory Multiplication sign Principal ideal domain Task (computing) Physical system
Context awareness Stapeldatei Process (computing) Computer animation State of matter Physical system
User interface Computer animation Information Inverse element Parameter (computer programming) Physical system
Computer animation Information Computer file Semiconductor memory Forcing (mathematics) Right angle Office suite Data structure Wireless LAN System call
Point (geometry) Type theory Process (computing) Computer animation Information Network topology Principal ideal domain Data structure Directory service Instance (computer science)
Particle system Process (computing) Computer animation Information Computer file Task (computing)
Logical constant Computer animation Information Term (mathematics) Multiplication sign Task (computing)
Process (computing) Befehlsprozessor Computer animation State of matter Model theory Online help Water vapor
Point (geometry) Computer animation Information
Computer animation Information
Point (geometry) Pulse (signal processing) Link (knot theory) State of matter Multiplication sign Mereology Coprocessor Dimensional analysis Number Bit rate Term (mathematics) Videoconferencing Energy level Physical system Interface (computing) Physical law System call Cognition Radical (chemistry) Computer animation Personal digital assistant Order (biology) Duality (mathematics) Normal (geometry) Summierbarkeit
Computer animation
so yes so once again welcome
here the next or lecture at the 1st and 12 of next lecture of of but I'm pretty sure it's going great because it's only just talking about leaders kindly body for is a supplements which all quite an interesting something of from my point of view and so it is 1 more thing before we started you like that of the talk or even if it is like don't believe but these for what is the pay problem to low to the front to the program of software and 1 Frost and III and this in the bag because it really helps those of 2 of 2 to organize and disease and so long yeah you have to talk about these these moments for integers theft and all i so sold home is a sincere good how many can hack and ISO 1 all here the for the other 9 so that you have the so what so this a talk is based on my experiments and experience this song my sysadmin don't like all on like somebody was being met in existence for quite a long time so this is this I hope this will help the system is well currently in the woods of trying to do something in kernel to do more of so I I can't give up with the Canadian company called at the end so constant talk so yeah so this
is the idea of this 1 I like to call examine basic investigation ontologies and some common issues and some tools courtesy and why why should the US assignments do you encounter the building of course to learn more about the system semantics the version even if presentation and brokers analysis this is going on all of the important nowadays it's more focus on SRE a the waltz kind of what flows integrating postmortem analysis and the thing in the world so so this is the 2 units and call in long must have seen some use a company or maybe in the shape of the land a lot seeing that the system is not responding and and the whole talk is based on those so I half the things to use how our students to answer before they with enough to live with white went and and because many times but you would end up would be a system if you face occasional issue but not all the time when times uh but let's talk about that more and so on and I'm just going to talk about stuff this we generally do so you basically identify with the of the claim is actually right whether it's a social order so this issue like this a user might be complaining of might actually start working the the system is on booking that doesn't mean that the system is told to do the basic of docking you do the basic of magic until being we don't set and see below the with issues system also this issue then you check out In whether what you see on the screen late if you hardly of any kind of user-provided counseling check of you if you're public review music to be in control of the KBM In return of period like somebody it's going to be a problem in all of the cells so it's it's really about such that different 1 person that has standard it said were on the screen I see below the from what is the issue if possible a little got to do differently you try to have see the Crab Pulsar quantity of whether the user but not accreditation use of it in the system using the reduced tool ifconfig loss of or you can check if there's a to tissue but to the system and if we accept all of this stuff and we identified that is a system issue of Hornby what do we do next I that it could be that could be cases of where the system is talking stop and you are able to come to the system a a lot of research funding at all you like you treat agency something on the screen or maybe nothing on the screen but it's not responding to anything uh maybe all men or you ever actually that took what would you do of the contrived Sasaki on so why society to society to on the said Carol part technology and mechanism which will allow you to send the keys generally discovered magic keys and it will allow the system to Don useful information it allow the the kernel to do some useful stuff like sink your 5 systems or even and you can that was the way that is useful and will come to that later mn but let's have to look at the you the so this is hard the
what I'm here how would you about all of these so it's almost like and yet OK this you for this so this is the process that you can go and sit environment again and it wouldn't on 1 standard which you can come in if you have access to a keyboard you can of time all CIS RQ and that led to key on so let me tie on under that to us in this society repressed would be echoing the betterment of according to the value for example a cool and the from society socket reveal it redundant memory information to your syslog the let's have a look what so you can see the
memory information is down here and what can we do with this I'll talk about this later on how to make sense of this and don't be alarmed at the bottom just
I'm on so we can also then the truck state information of all the processors that's cute couple seconds
what we can see the process
trace information from you can see the process name you can see the US state of process the the the process sleeping or and and optimal sleep just the state court which is inviolable status Ostade then you can see the court part in which the process that
the important process was excluding what to do
the core but I generally what if i wanna try understanding what is going on with
this I just look at the count functions this
other kind of functions and the may be
going the county and so I have already learned see dots here a lot of people who are all yet so different however this is good of so I
have already done syntax and it and basically created a death sit that information from so that the texture it'll make me think and haven't for examining just going this can be a force right in this for so again this Lord apply to the
court I can look for like you I wanna
see other instances of according this eastern look
a lot of people who do of
cis scope but I don't use it there's an article on the so I don't have to search tool which of course so
that is RQ around the talk aboard I mean how this can be more
useful would be finding the arguments which are lost on by 1 function to honor than that being listing come on finding documents for of function and you come to that later the school and continue so after the
a wheelchair cysts long of course you know 1 of the stages you know 1 broke kernel of the new circuses stacked up I hope it really helps to start and solving a system by even if you how you fancy metric system definitely tried to install Systat include that of in your images so I'm pretty sure most
of the people enormous but on this show so this . includes sort which of you can have a look at the system log information so sort of our system of law information here and you can see that
the Lord Wilson userspace
lots of systems based on your way to all this stuff this is all stuff which we
generally do not against memory supplied
like this if in have more questions you can ask after the talk many of these all the
of when we have here I'm pretty sure that mean many of you how interface that someone standards and help coordinate the ankle 1 time of the 1st assignment so we and and memory done all the other axis on the topic would be in court on later to now let's just
see if you stuff like like banning sent a catalyst conditions which you might see I 1 you must have seen that any what life so this is a normal panicked too many or toss it for a reason because it's pretty straightforward and easy to explain so this is a general grant of any agency the the file the source code and in-line with the soul of this band it's only happened in the that is a condition called above all the internal source and if the accord and the being the so this can this band hits and you can see the sort of idea I just from just showing these just to get you a bit more of formulas for the structural of panic so the next time you have a look at it you can make more sense out of the so this was a seat you on which was the executing at that time and the more you slaughtered In the system during the time of issue in conceived the particle module is having any kind of fracturing of proprietary or force law again see this but you can see the PID of the process which was running during the time but it also scan or bandit economy did it process behavior can the bank but the process from user space cannot panic and if that happens something problem the problem is in the then you you can see the coalition against the so the amount to standard that any proprietary more you or stop and this is the most important thing the pure starting with this so if you just want to without going through these you just who this this is the intersection point of this loss of function which are also being executed when the Spanish happened so uh the 64 bits you can find the RIP introduced CUA I'll skip the whole order history part here and go to the call tracing so this is the this the portrays like the called Husain in this sort you 3rd state or put you can see that the system core in function and it is that most of the of all probably all of cis system correlated can also like consist underscore prime sources and the core of the system going so in this case the 2 of processes that are trying to do something and also they Spanish-English which cost issue not get into how could be but it can bind that's a big big topic and I'm not 100 per cent it's on the uh qualified to do that but I can actually pointers on how to do that and I have some specific knowledge important specific part of subsystems but definitely if you have any questions i can try to point you to people upon you to resources but so this is a salt locked up and again why would it be get many happen I can't bend happens when you can look at things that at this point of time I can't properly uh the goal of the system carried my own thing that you have and this point of time I cannot so this before and continuing this might posted Dactyls in that case and that is when a chemical spending that is the standard definition that the Lord of different but noted below the but this is the baseline so what the text is a block of the solve flock oneself local doesn't need doorways and cost the system unusable soft local usually happens then Carol tries to every process from the CPU but it is not able to so it it's continuously arriving for 10 seconds I sort hill against see the the inception point of IP because there's that that the systems and you can see this in the pattern is saying that the again see the CPU against see of the industry information you can see the portrays and kill 1 thing you can see again see that some of this court is coming from module 1 specific more new so I was this specific orders coming from this morning this is also kind of useful information the notion of an example of how the task of hand does what happens then it also so he's in the state for more than 120 seconds what is the state of the state is up on and shopkeepers leap that they basically happens when the process is waiting on how high you some most of the cases of that can be found other cases still mostly you so Hill the process is waiting on distant from more than 120 seconds you can this this can this behavior in in some cases can be an expert to begin the process might be that supposed to be reading link running being on the state for a long time but that can be go cases in that case you can this December the hung task in dust and the solution doesn't make the system usable but many cases it can hit also again see the process from the the process state and the call trace you can try to a little girl in accordance with a functions 1 by 1 and see how the the course false so the result of invented sure that all of assignments helps you note of nonlinear of she'll I have chosen this the specific heat goes from this week message because this is from within your government and I don't have to explain what's nowadays I would like to explain all of new mom on the new of correlated changes of all the stuff she'll went around you can see them it is dumping this is the same information that is done by the Sasaki about in limited and so the pattern that it would look the same cost to have different information but the the pattern is all the same you can see that the you can see the number of active bidders number number of inactive status active and inactive because basically means that I can use this it in his corner lot you means recently used it is used to identify whether it particle a process is being in it but the base where the page is commonly used on all of you can think that 1 would think that it is active at a mean of being directed list and if it can't 1 to free a particle of pleaded in fast would inactive before it freezes so you have the active and inactive pages the 180 pages 30 but the is that of in the in the moment it's how change information the tunnel prechamber right that is the basis the photo of a book that limits of an in-flight it is becoming computed and back to the system and in this constant police a specific stuff and a star so this unstable tree each of how of basis which you can and that of course is free money and slab slab is basically a kind of objects which are used to define predefine the kernel so that you get continues so the Bates allocations become contiguous so it so it usually how the Bentley 11 information it and how of the towel that came out of here objects you can check our slapped all money your system to see the slab information map is the map status so it's basically here in pages which does have a file that backup and then based in the interest
of it's just have all the information on the pages on in the memory out there so this is this have ligand list of our biggest all aren't really it's time to show you that all the bidders of located and then you see the difference so once you CDMA Norman Heino is a 32 bit system that's why you see or do you mean Norman Heino stuff on so 1 thing why did this system panic because it does have a lot of memory it does how almost of 3 pages it does help conserve room on the comedy any idea why would this of system based on performance when does the system phase out of money yeah so to yeah yeah think this is this year in there when the system normally when the system right so the only way when so near and also found yet so the question is from me in focuses on you want to be my question of his question the answer because of this all his life all of these are so on let's continue on I'll tell you why and the system the system was for the better because of the songs but because it's own that's what want this source of indium it traditionally behind the and normal and high and it's a bit complicated topic but no long term that when the the many ways can only access of some of the newest can only access the DMA so generally axis of 2 I think of 16 yes 16 from and so on so for those devices of the Amazon was introduced and research than our general mostly most enormous on and higher is used for mapping most of the memory because the the majority have come in Lake theoretically can only have 4 GB of right so it's a bit more complicated topic but basically we have different SOMs laid out the main normal and high low and in this case the each Saunders how few what remarks 1 is free minutes and law noting is when the Freemen goes below mean that is when an order from what happens in the end that is this reason any of happens so when they're the female goals below the goal it is the kind of lose very actively but try to reclaim only by using your really fresh using of what role the current demon governments running kernel demons to try to reclaim only and and to try to reclaim memory and of the free is high so in this case even though there was high high on the solving of free memory enormous In hadn't been way so this was like a historical problem with their the systems and our is don't have to worry about it I just used as the x fermented which system so that I can explain that that that is something called songs in our memory in which a more then we have but you look at a when you look at a basically kind of show you sure that cannot In each songs how much contiguous memory is already so if you see the if more memory is allocated in 4 KB the chance of the the system being in a system only being highly fragmented status high so when uh the the more memories human the the with a lot of time that means that the is less fragment so so that is 1 more thing so you know some of the obligations probably need contiguous memory but lost because when we so when it tries to allocate got elected because visible money it can how big the location of innocence afflicted and yet you have the stops information than the how the process we just can't answer for killing a process of normally a functional and is used in this use different on the logic of lake except of logics are the to allocate the points to it but different processing the system works under if a process is nice to it would be even less ability to compete in a process is a little problem of on very different if it is I'm but all the time so it is getting in this last for the badness value is based on that and I'm currently in in the current going be held on score and ADJ parameters the its can kind of tuned this behavior for each of the eighties and to make sure part some if you don't want your may still to be begin you can uh boss of a value to those and score in the broke the anyone score and the you can lead to serious than it or minus 17 or something then it will make sure that the link it happens of your fellow process a can what the what then the house of Norman hands on like can is a bad moment eliminates many abstract on what he'll just I have just listed of the cognitive issues of like that condition checked exceptions exceptions for field detection of collection in that stuff so if you find anything like something like mission to even so you can have a look at why Logan's you know from an probably contact the vendor if it said that much of system if it is the height of its high way so it's a medium to high voice of of the topic the high was on the remain the same the many things happen if it happens in a little machine it's probably about then that can be CPU memory of you play sessions which can cause a half so these In or handouts these are also kind of a resource allocation issues but you can also use sort tools like sought to identify issues just cost by high utilization of resources let's get to the input of it so off the end goal is a form a dummy immunogold we need it got than there and traditional long been back those already and number and this demo remember 1 and then was dumping them memory of all in the book and this dumb was jumping in lock in this country genome is capable of dumping it and so now yeah can and can't get misconfigured need get here if you can can already done when that is a bank Kevin bounds of the look to the uh the the the more information of the memory of the system when that that's a apparent so if you specifically wanted the book 1 of the previous issue in detail if you don't understand from the screen if you don't understand you can initially band gonna I'm not talking awarded tenure and this PanImages and this so this thing that if you have a good initial if you want to avoid like an ongoing issue if you and have a deeper investigation you can pass this parameter so when that is a sort local water and all in all hung task of the system so that I can number it the angle and you can also toward society see all of previously I don't
system i 3rd information so that if and when you see this panic systems are not going to do that and that will be done in idiom core and if you do that by if Walker latch so but
talking so let's have a what's attack for so let's have it you can look at me in court so I have to be in court you which I have so this is really called I have learned from the system and full of unlacing Indian court I need it that it commandment to go crashed clashes that basically a wrapper around the GDP to which you probably know and this the minutes I have extracted from the general heightened debugging for packets which does have enabled symbols of unlike you'll be in minutes filing new of slashed with so you can
see the chief information that will give you the basics of basic information on the canon that when it crashed long-term back so it's a doctrinal system the time of the panic
so you can see that the panic yours cost places are cute don't you can see
the uh general release information also we can see the prosthesis which are running
in the system at time of issue but I can see the PID information you can see that task you can see the state of different processes all used by processes the i in the sleeping but if it is you and if that is
a UN that is under conversely if I want to see the trace of 1 particular
fossils they can still the the idea the handling the interesting thing would be the memory state during the time of
issue the agency friend and the system was hung on Princeton Watson panic state what was happening in the system I know so this so this was the process which actually banned the system by this or some context so you can see that it is biased because I run the command from batch identical see rocks Osaka Journal from bunch we can see that the Soviet was right in the
Proc of 5 systems now what
is the inverse of the 1st user interface with the found between different 5 systems and the current so she'll watch all I would like
to you so I was trying to demonstrate how to find an argument which is passed so you can see the stock information like this from 40 so this is done so
company extract all this call trace we can see that officers after cis right it's going to be a force right and extract information here from this doesn't make sense
these looking around so let's see what does the 1st right how so this is the coordinating the school to the
source of possible we can see that it does have a structure as struct struct wireless for starting so I'm going to pass the abstract file and going to use that memory
can so you can see the structure of the world what the using the structure so she begins a new IT
the PID struggle with all this information what's interesting show used
the gentry then trees the and the stories of place of you directory structure for instance that's just 1 job entry so let's check stopped directory tree and see what think the what what type of point so you can see the name so I
this just looking for this you can see the in the in the gentry that's how the information of the obviously nor that I want to suck it triggers made more to dissect this but it looks like this particle process we can see that this was the name of the heroes so
they are emotionally can't see that's named after
the file it has been accessed and the thing which you can do use
probably look at the task the status quo apart from a few salient
start for reason for task you can see the
task related information times term we can do a lot from this work on the task related information is curious about so the might of Sauron the top all the other set so yeah this is basically what I was planning to go fighting I was so fast yeah so can you in constants we had lot of the
all yeah uh so yeah let's not see
how the help me how q so thank you
just show you some water processes which will running in each CPU and doing that it is somewhat what of course is it's learning model process which anonymous state and the current state coming process starts with an then what instantly how what I am
familiar with yeah but the information there of course so that we
have a small and information is also there so if you
want to see that the story point point 5 so but for we have to get it from the start portion there so you can get it off in I have to check whether it's thing the it's summer school more room what exactly what yeah it was small we can see you can bottle that want information about what this stuff like that
why you see a sister neural when information yeah any other questions
yeah I think of as being made by the way that can be something to get it all the way back to where I had this so that the so they come from the yeah yeah it's on this that we had all it I the end of the 5 minutes of the problem audience with but
any other questions yes the last year so what about the you know the it was the and a lot more than the sum of the last their so you're talking aboard to on the
if you have a you 1 or so that on 1 thing if you were dimension if you want to run any of the society you probably to such to what determines the Court of going around the following and and if this is not the case so 1 1 thing where you can do probably is it OK so the Cassini seal having uh AUI interface and if you face a hand how you recover but if it is found if you're having this issue and if it is an ongoing issue and you suspect that it is like a large number of the state processes Lord of you all if you want this find out what is happening and what's this video introduces Lord already issue that is a solution that is to work or how hand-in-hand what's hand what's what had much does is in the same cell death and watch it with more until the lord knowledge of order of is submitted to piece of the normal rates doesn't mean it is that's a problem reduce the cognition based on the number of processors and uninterpretable sleepless you basically upstate understood pulses so if you got out it is Fort Sumner level 10 of the hand what's to detect that and it will automatically run society you be like rituals that you want confident you can compute this in hand watched and again do that is 1 possibility and the thing is that use their own talents and how about some of them at the role like that there's 2 of and these these people in most all seen in my lab still uh Eve of that is a problem which would affect the interests of this non-muscle into its center might hint of sonic commanding all time it will generally simpler done I and the me what still get 0 and if you consider that in way what stock should banning the current it was considered by so that can also have its own can some of the what's the city in a few problem is that you way I think it's but it probably taken try to correct the threats that can you do and you can see yeah you're you're saying you have a l the holiday dual system which use where they mostly for you 1 links in almost something the kind of and in the and so on all the profit of the kind of these users all and you know was the the the the personal and in the that all the book in the 1st of all they was to know what we so it's out and kind of like they were the general term lighting Uganda's still think or if 2 and part which of the refresh the terminal weekly for like it's a norm of the nothing is more in depth with the and and I know it all the rate of OK so the the point of all the rest of the ask you for your so it the that and the use of the when you can after you load a call the system you can take the loss and see what was happening in the we can find something from the laws and based on that you can continuing investigation when there is no 1 way to do things you just need to figure out ways based on what you have that and thing else a for answering the question and this good but