AV-Portal 3.23.3 (4dfb8a34932102951b25870966c61d06d6b97156)

Kexec/Kdump under the hood

Video in TIB AV-Portal: Kexec/Kdump under the hood

Formal Metadata

Kexec/Kdump under the hood
How to debug your crashed system
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Kdump is a vital tool for debugging severe kernel crashes, especially if the failure can't be reproduced easily or an direct access to the system is not possible. When an sever error happens in the kernel, a new crash kernel get loaded which saves the memory of the crashed system. These dump can be used to analyze the state of the machine and hopefully give insights on what has happened. This talks will dive into the internals of kexec and kdump. How the crash kernel get set-up, how it's execution get triggered. We will also look into kexec-tool, the user-space part needed to set up a system to use kdump. Where necessary, the architectural specific details will be explained by looking at the arm64 implementation. This talk is thought for people who want to have an insight into how kdump is working. Kdump is a vital tool for debugging severe kernel crashes, especially if the failure can't be reproduced easily or an direct access to the system is not possible. When an sever error happens in the kernel, a new crash kernel get loaded which saves the memory of the crashed system. These dump can be used to analyze the state of the machine and hopefully give insights on what has happened. This talks will dive into the internals of kexec and kdump. How the crash kernel get set-up, how it's execution get triggered. We will also look into kexec-tool, the user-space part needed to set up a system to use kdump. Where necessary, the architectural specific details will be explained by looking at the arm64 implementation. This talk is thought for people who want to have an insight into how kdump is working.

Related Material

Enterprise architecture Computer font Distribution (mathematics) Arm Demo (music) Software developer Bit Software maintenance Mereology Twitter Software maintenance Product (business) Kernel (computing) Personal digital assistant Kernel (computing) Software Personal digital assistant Core dump System programming Spacetime Physical system Spacetime
Presentation of a group Multiplication sign Motion capture Virtual machine Mereology Graph coloring Product (business) Revision control Workload Latent heat Crash (computing) Goodness of fit Blog Read-only memory Semiconductor memory Kernel (computing) Personal digital assistant Computer hardware Core dump Video game console Arrow of time Booting Firmware Physical system Computer architecture Boss Corporation Serial port Distribution (mathematics) Arm Software developer Debugger Core dump Motion capture Machine code Virtual machine Kernel (computing) Personal digital assistant Crash (computing) System programming Physical system Firmware Booting
Implementation Structural load Motion capture Parameter (computer programming) Mereology Number Crash (computing) Atomic number Computer configuration Kernel (computing) Core dump Spacetime Firmware Booting Physical system Structural load Computer program Core dump Mereology Motion capture Latent heat Kernel (computing) Computer configuration Repository (publishing) System programming CAPTCHA Physical system Fingerprint Spacetime Booting
Ferry Corsten State of matter Principle of maximum entropy Parameter (computer programming) Mereology Computer programming Area Semiconductor memory Different (Kate Ryan album) Kernel (computing) Befehlsprozessor Core dump Flag Information Physical system Area Email Structural load Web page Range (statistics) Bit Motion capture Befehlsprozessor Crash (computing) Quadrilateral Buffer solution System programming Physical system Spacetime Booting Point (geometry) Web page Principal ideal domain Motion capture Product (business) 2 (number) Crash (computing) Read-only memory Data structure Address space Metre Information State of matter Core dump Usability Semiconductor memory Uniform resource locator Kernel (computing) Pointer (computer programming) Network topology Flag
Point (geometry) Functional (mathematics) Structural load Ferry Corsten Execution unit Range (statistics) Motion capture PowerPC Streaming media Parameter (computer programming) Mereology Computer programming Product (business) Usability Number Crash (computing) Network topology Semiconductor memory Computer configuration Kernel (computing) Core dump Information Booting Address space Physical system Exception handling Area Email Arm Information Assembly language Sigma-algebra Visual system Structural load Usability Line (geometry) Complex number Sequence System call Number Kernel (computing) Hash function Network topology System programming Hydraulic jump Spacetime Firmware Address space
System call Multiplication sign Execution unit Mereology Machine code Computer programming Area Medical imaging Semiconductor memory Kernel (computing) Befehlsprozessor Core dump Flag Local ring Physical system Area Email Structural load Web page Range (statistics) Bit Maxima and minima Motion capture Control flow Sequence Befehlsprozessor Process (computing) Crash (computing) Buffer solution System programming Physical system Reading (process) Spacetime Booting Point (geometry) Web page Slide rule Game controller Functional (mathematics) Patch (Unix) Computer-generated imagery Virtual machine Motion capture Content (media) Product (business) Number Crash (computing) Cache (computing) Read-only memory Network topology Rootkit Spacetime Data structure Booting Firmware Address space Information Validity (statistics) Voltmeter Machine code Semiconductor memory Limit (category theory) System call Cache (computing) Exact sequence Kernel (computing) Pointer (computer programming) Personal digital assistant Rootkit Hydraulic jump Flag
Web page Scripting language Distribution (mathematics) Multiplication sign Execution unit Motion capture Mereology Stack (abstract data type) Approximation Food energy Computer programming Product (business) Web 2.0 Read-only memory Semiconductor memory Core dump File system Data storage device Configuration space Physical system Scripting language Area Module (mathematics) Distribution (mathematics) Plastikkarte Core dump Mereology Motion capture File Transfer Protocol Kernel (computing) Software Data storage device System programming Configuration space MiniDisc Website Physical system
Web page Laptop Point (geometry) System call Distribution (mathematics) Computer-generated imagery Virtual machine Parameter (computer programming) Product (business) Crash (computing) Core dump Booting Physical system Chi-squared distribution Newton's law of universal gravitation Module (mathematics) Scripting language Execution unit Email Arm Demo (music) Structural load Mereology Ultraviolet photoelectron spectroscopy Graphical user interface Kernel (computing) Befehlsprozessor System programming Configuration space Right angle
Mathematics Crash (computing) Kernel (computing) Different (Kate Ryan album) Computer-generated imagery Ultraviolet photoelectron spectroscopy Physical system
Wiener filter Cone penetration test View (database) Computer file Programmable read-only memory Motion capture Parameter (computer programming) Limit (category theory) Kernel (computing) Read-only memory Kernel (computing) Befehlsprozessor Revision control Video game console Physical system Firmware
Ocean current Point (geometry) Empennage Random number Vapor pressure Ferry Corsten Execution unit Plastikkarte Mereology Emulation Product (business) Magnetic stripe card Pointer (computer programming) Crash (computing) Virtual reality Read-only memory Semiconductor memory Kernel (computing) Befehlsprozessor Rootkit Proxy server Arc (geometry) Physical system Area Game controller Public key certificate Mapping View (database) Computer file Keyboard shortcut Computer network Grand Unified Theory Word Pointer (computer programming) Befehlsprozessor Linker (computing) Personal digital assistant Interface (computing) Multimedia Computing platform Daylight saving time Session Initiation Protocol Reverse engineering
Random number Ferry Corsten Range (statistics) Source code Motion capture Machine code Emulation Geometry Pointer (computer programming) Crash (computing) Read-only memory Semiconductor memory Befehlsprozessor Kernel (computing) Core dump Information Physical system Email View (database) Computer file Bit Kernel (computing) Linker (computing) Personal digital assistant Computer hardware Revision control Personal area network Exception handling Electric current
Point (geometry) Area Source code Email Information Multiplication sign Point (geometry) Source code Motion capture Core dump Motion capture Computer programming Area Product (business) Blog Read-only memory Semiconductor memory Kernel (computing) System programming MiniDisc Information Physical system Speicheradresse Physical system
Module (mathematics) Mathematics Kernel (computing) Multiplication sign System programming Motion capture Freeware Food energy Product (business)
System programming
okay hello everybody to the
before lunch I will try to explain you a bit the internals of kxa kdump so my
name is Matias Braga I've worked for Souza and I work mainly on arm 64 so I work for the enterprise product of Souza and are 64 and also contribute to openSUSE and as part of this work I was working on kxa km forearm 64 and then I want to explain you a bit what I learned apart from that I'm a maintainer of mediatek on the system mediatek system and chips in the linux kernel and i do a lot of more stuff ok what would we talk about today so I will quickly introduce the use cases that exist for a k exec a dump and then we will dive into the internals of user space in kernel space I will just describe a bit what support is given in openSUSE and which is for shown as a distribution as well and then we will have a quick demo and QA ok so
basically there are three use cases first this for me is like the most important one and then for sure is what why this in the first place was developed that is to debug a system so it was a for example we have a lot of customers so they have hardware and then we don't have maybe don't have all this Hardware in-house or they are running some legacy code on their on their systems that we don't get a good reproducer when their system crashes there might be that we don't have good locks because Colonel Chris is so badly that looks actually not usable and then for this case we can use K dump to create a dump file which then afterwards you can inspect to see what actually happened and why the system crashed as an excuse cases to put a new kernel without rebooting the system so that's for me when I first heard it was a bit weird but the idea behind this is that when you have for example assist a really big machine which will take a long time to to put up because this from where has to enumerate all the hard way etc or you have a system which is pre pre production system and even the a firmware is on the development and you have some beta version of the firmware then you can have packs in the firmware and so it might not be sure that every time you boot the system the system comes up and so if you work on some kernel stuff then it might be useful to just reboot the color from the kernel and skip all the from we're all different various initialization all the boss also talked yesterday about putting using Linux to boot Linux and that is exactly what is done with IBM s/390 which has some really big machines and they have like a boot loader a Linux and then they KX akin to the distribution kernel and throughout the talk I will talk about the first use case which is debugging a system the other two use cases well they're basically the same and technically and i will just i won't i won't go into detail about this and then there are k x ik k dump has like some generic infrastructure and but there are some architecture specific parts and where the work would go deeper into say architecture specific parts i will talk about arm 64 because that's what I know best and so the different names for the production system so the idea when you plug a debugger system you have a production system which your production workload running and some really bit back in the kernel happens and then a capture system get started there different names for this capture system crash system panic system I will try to keep with capture system or capture kernel in this presentation - yeah to not confuse too much so how does it look like I made a little crappy graphic so you imagine this is see and you can see this you're right yeah imagine this is a memory of your of your system so you have somewhere in memory your production kernel which is running Jewish which shows this arrow and there's any of your system RAM and when the production kernel crashes then you're the capture kernel get started in the capture kernel then takes all the memory and the kernel of the production system and creates a Tom file so which
parts are involved in this so there's a
user's user space part which is called kxx tools that is basically used to prepare the capture system then of course the kernel itself that and that needs to execute and capture the kernel and crash and there are some other user space tools like make dumb file which is of and which is used to create the actual dump file from the capture system and then to inspect system you can use crash or crash patent which is a implementation effort and by Susa to those that you can use patent scripting to and to inspect atoms and then there are some districts to to make it easier to set things up okay so kxx tools there are different tools in the in the repository but the most important one is called KX ik those is command line what that actually does it loads a kernel and an init Rd and reuses a put parameter of the kernel of your system and then you can with dash e execute this new kernel and that will K X again the new kernel and what it what it internally thus it calls reboot reboot with some magic number and this magic number then the kernel recognizes and then says okay I don't have to reboot into the firmware but you reboot into the loaded kernel and there's also now confusion starts so you can also load a captcha kernel which is done with - P which stands for Pennock and you can unload the kernels and that there are some specific options which not relevant first now
okay let's have a look Untersee understood how this okay exact tools actually works so we remember that the
capture colonel will make a dump file out of of the of the production system though the question is how does he do it and how does he do it and what does he need to know so he needs to know where the actual capture colonel is so that he can load the capture colonel when the production system crashes and we must know where as a usable memory is for the capture colonel because we can't use just a memory that your production system uses because you would override the memory and maybe we'll will overwrite the hints you need to find out what really happened and you need to know where the user space of the of your capture system is and you only need to know where the production colonel and memory can be found so kxx tools okay X second general what it does when you put your production system you pass a kernel parameter which is called crash kernel which is create and reserved memory area so if this down here these books is your memory you have then the great part is in the reserved memory area which is in lead not used by the by the production system and it can be a bit tricky to do so because you don't want to have this memory area too big because it will be taken out of your production system but you don't want to have it to be too small because then maybe your capture system won't start and you can see there are like different segments in the in this reserved memory area and that these are called kxx segments and this is a data structure which is shared between the user space in the kernel and it basically describes is used in the following way when you for example load the kernel with KX ik tools what KX ik tools does it allocates a buffer and on this address on this pointer with the size of puffs size and then copies the kernel into this buffer and then it searches in the reserved memory area and a whole wait can actually later copy the kernel to and points this void main void main pointer to this location so that would be to here and that's it what it does was care and with the kernel and in it Rd and for the elf core headers a device tree plop in the purgatory I will explain now what what these are and what they are doing so the elf core header is like the core part of kdump so that capture system knows where the memory of the production system is so this an elf elf header which has different program headers which then points to Z and points into the memory of the production system so remember this elf Co header lives in the reserved memory area and the first program headers for each CPU which is which are called crash notes crash notes is a crash note is a small part of memory in the production system that is reserved for every CPUs is that when the CPU crashes it can write the richest estate P etcetera in there so that you can later find out in which state every CPU was when the crash happened and from user space you can read system and this system is file and then to find out for every CPU were this memory where this memory is and K exact tools what it does it actually writes the address in the in the program header then there's another and important data structure which is called vm core info this data is a data structure also in the production kernel which holds all the information about about the data structures of the kernel so for example the page size of your system see offset of flags and some structs et cetera you will need this information later to actually understand in your dumpfile how the car looks like and this file can also be read from the back exit tools through this sisyphus command and it has the address in the size which gets also copied in here and yeah that's it and last but not least you have some memory in the kernel itself for this and what kxf tools does it parses proc io mem where you can find the system system rom parts and we're in which part of the system ram the colonel gets loaded and takes his address as well and put it in in some headers and this structure is then later used to create a file in the capture system that's called proc BM core from which you can create certain
file then so next segment is see device tree or device tree blob and who knows what a device tree is quite some quite and quite some people nice so device tree is like C equivalent to a CPI which is used by arm and by by a PowerPC arm 64 actually has the possibility to put this device 3 always a CPI and so to come over this confusion what what the system does is when the production system boots up the if we stop creates small and flat that you flatten device tree which just holds some really basic information about the UEFI and about where the unit Rd can be found and the and they put parameters and the K exact tools XK exact tool reads this file and then updates the unit area and it earlier in it Artie and points it to the unit Rd in the in the reserved memory area then it's a it's a value for the elf Co header which points to this segment and the usable memory range which would be all the rest of your reserved memory area which is not used by the K xx sequence which is then the memory that you that your capture system can actually use to boot up and and do all the stuff you wanted to do in yeah and the last second segment we have is a purgatory so the pro guitar decides over a heaven or hell and heaven of course is that you are lucky that you can boot your your capture kernel to actually create a dump and hell is that your system is so broken that you can't even put that boot that capture system and how does it do that well the purgatory when getting set up by Kay exact tools it holds the edges of the other four and kxx sequence and when the purgatory is like the entry point after crash first what it first does it checks if the edges are correct and because if not then you go you can't you can't be sure that the capture system you put up really holds the information that's valid for example imagine that you go ahead is da is broken somehow and points somewhere else then the information you have is not not correct when you won't be able to reliable the debug see system there's also a command option for kxx tools to ignore these checks which is nice because then you can you go always to heaven yeah okay what the product Ori actually does for arm 64 it looks like this I hope you can read it more or less so this is the assembler it's not the whole file I I deleted some some lines but basically what it does it creates it has a small stick then it calls purgatory which is a see function which the only thing it does actually is check the hashes of the different segments and it it's n loads a kernel in a the kernel address in the rig and I register the device tree address in the register and boots the kernel and this part is just like the normal way to boot arm 64 so nothing nothing special in here and K except tool was it asked what it does with C's file is it updates the kernel address in the device tree address to point actually to the kernel on the device tree in the reserved memory area yeah so then in the end our knowledge stream and call it like this looks like this so the pro guitar which is like the first first to be executed after crash he knows where the kernel and the device tree is it starts a kernel and the kernel can read from the register the device tree and the device tree holds the information of the elf core header seen it already and the usable memory it can use so that's good so and now we have set up all these segments in kxx tools and then we have to pass this to the to your pro to the production kernel who said he can actually prepare for a crash to actually start a program in the capture system so this is done using the K exit load system call there's also another called K xx file load which I won't explain here because it works a bit different because it doesn't most of the stuff we have seen here in the kernel space so it passes the entry point of the program to the kernel and the addresses of the Sigma and bolts in an array of the sequence and the number of segments it exists and that's all
what we have to do in user space and now we have passed all this information to the kernel and now I will talk a bit about what the kernel actually does so
in the kernel internals we will look into the three parts so there's a first part is and that the production kernel pre-press the capture system and we through the akx it loads this call and then we will have a look what actually happens when your when your production system crashes and and then we will see what happens when the kernel boots up so when you load your capture system what they what the kernel actually does it just check that your root because you don't want to any you into have any user to load a new capture system on your on your on your machine it checks some flex that the basically basically means that you are not allowed to load another capture system if you ever have already one loaded so you have to unload it first and then you and then load it the reason behind this is that if you in the updating of the capture system you get a crash then everything is lost and then it it checks a segment number so you have like am limit which is I think 16 of segments numbers you can use so we used for that we are totally fine here and it then creates a structure what allocates a structure in the kernel which is called K image and this holds the KX ik sequence and the entry point of the program and it allocates and holds a control page which is later used to actually load the purgatory we will see that in a minute well then they will Colonel will do some sanity checks so basically checks at the and let's see sequence not overlapping because it would mean that something goes really wrong that they are paid aligned and and that they are in the reserved memory area and it also checks set some em size is bigger or the same as the buffer size the reason behind this is that the memory and there that you use in the reserved memory areas patch aligned and this buffer is created by malloc and user space so it's not pitch aligned so this can be bigger and can't be smaller and last but not least it checks that all the memory you need by your kxx sequence is at least well is a maximum as big as half of the of the RAM that you are available on your system so when that's all okay then it is calls copy from user and it copies this buffer with the buffer size and into the address of each segment in the river in the reserved memory area and then it clears Apte valid bit for these segments for the pages of these segments which basically means that it's let's see mmm you won't access it so that's that's what it does well last thing and it's not good in the slides but it actually sets a flag to say that we have our capture system loaded and then when they currently crashes and it realizes that that capture system is loaded and then it starts to disable the I accuse in the CPU registers and it will write into the VM core data structure and all there's a time of the crash so that we have when you have a crash dump you can also read out when the system actually crashed and then it will send a signal to all the other process process was a new system to shut down but to shut down using crash the CPU crash stop which is special and handler that basically writes all the CPU register or the CPU information into the crash notes as we've seen beforehand and disables I accuse and then tells a firmware to actually shut down the CPU so that in the end you have only one CPU running or that is what you hope so he checks if only once you be running the disappear that crashed if this is not the case then you have a problem because it can then be that a crash system when their capture system volt won't bond put a boot up correctly but he just wrote the warning and goes on because you already crashed your production systems oh yeah what can you lose yeah then it uses a relocation controller some relocation code that copy will be copied in the control page this coach shuts down MMU and disabled the caches and then it checks if you have to relocate the colonel that's the case if you just want to boot a new kernel and so that's not not the case here and then it jumps to the purgatory we have seen what the booger program sorry we have seen what the purgatory does beforehand so check ck exact sequence and then load the kernel and the load they loaded kernel can then read this device three entries which point to this kxx sequence to to know which are where and how BIG's your memory is it uses and where you can find the elf Co header and where you can find the unit ID and then can start seeing it ID and he reserves some memory where he copies and the elf Carrera so remember the fqo header only holds pointers three copies of pointers and not the dereference of the pointers and it creates a file that's called proc vm core which is a file that you can then use to create a dump file and this file has a special a special read function which dereferences the point us so that when you read this file you actually read the memory here and not just the address okay so
distribution part so does that set this all up it's a bit difficult because apart from deciding how BIG's your reserved memory area should PU would need to have an especial energy and it depends on for example where you want to save your dump file in your capture and your capture system if you want to save it like for example via FTP to an FTP server then you will need you will need all the drivers for your for your for your network card you will need an Network stack and I knew only their FTP programs in unit Rd and or when your you want to save it to disk which is this which is not what you should do because you could could have a crush in the file system and then you can't write it to disk but you can do this and you will need a file system more kernel module and you in et cetera so there are there are tools to help you to that let up so that you don't need like a full-blown in it or D which will just waste memory for it will just waste memory that you don't really need and normally this distribution parts also have a way to say that you automatically as mega and store the dump automatically somewhere so you don't have to do any manual intervention and then reboot the systems of that at least the system comes up again and you without downtime is minimal while you can then investigate what really happened and it Suzanne openSUSE we use a program that's called kdump voice capital K which is a Swiss Army knife setting up okay dump so it has basically two parts so that's one part for the production system which is some very good scripts to create the unit Rd and then bash scripts to load the capture system which basically calls K xx - LP with the kernel in it Rd cetera et cetera and there's a tool to approximate the size of the reserved memory area so that you don't have to do and try a narrow in on the capture system site they can configure eight how to create the dump this is done mainly by using make make dump file which where you can decide for example if you want to add free pages to your dump or if you care about pages that are all 0 is set around because if you just copy your all your memory that might be a really really huge time file in there might be parts of the memory that you're not interested and you can also define web where these don't get stored so this is a configuration file and if you're really lazy and you don't want to understand this configuration file you
can use a tool that's called yes - which is a openSUSE or souza configuration tool for the system and there's a module called kdump and you can see it here this is it looks nicer if you started from from a GUI right this is from serially console so you can decide for example if you want to compress the dump and which pages you want to include into sit down and you can also for example send up some email notifications so you get an email hey and one of your system badly crashed and so you better have a look ok so I will give a quick demo I
hope you can do this because it's black and I wasn't able to change this so what I am doing here is I'm starting a virtual machine which qmu this is arm 64 machine also my laptop is not arm 64 and we can then have a look for example about how many CPUs so we have 2 CPUs we have one point eight gigabyte of RAM we can have look on this script so I wrote a small script malice script it's just a one-liner right and that actually loads loads a kernel image with an inner daddy it ignores it checks because we want to be fast because we all want to go to lunch afterwards and it depends some command-line options so the command-line options here is that you just start one CPU and said you reset the devices and one thing that I wanted to show you can see in the production system in the command line and the kernel boot parameters though that we have actually and 180 megabytes of RAM reserved for for the crash system so if we now love this we can so be you have also never like this whoa so you can see there the
different CSS files but you can see if they if I could yeah if tress system is loaded or if a normal kernel is loaded for just rebooting the kernel and not caring about crashes and the size of this so it should be loaded by now because we and we loaded it yes great so what we can do now is we we can write we
can outright see well you see - Susur ku
RQ trigger what it does it create it crashes artificially your kernel and this used to just check out that this actually works so what we expect now that the capture system comes up and that we can then do some fancy stuff so we can see here yes you can see here okay so we took out a
crash su through sis sis RQ which is a current little pointer dereference and
robert habla word comes from and it's in this okay we're stopping all the secondary cpus that was just one and then it starts a crest and the crush colonel and then you can see here but then capture colonel and then you can say here how they can capture colonel comes up so that's the first step throw really good we can now see proxy pu in food that should only have one CPU because we passed this mac cpu equals one that's the case so we would expect we had one point eight megabytes gigabytes of ram in the production system that we would expect like 100 80 megabytes now it's actually 147 megabytes because you have some part from the reverse of memory area which is used by the k exit signals and what else can we do we can do we can we can have a look on the device strip bind device three blob that gets created and that got updated by kxx tools so you can see we have here the wavy map and we have here the unit Rd and up here we have
they use every memory range that the K exit tools other than the elf Co header and where this can be found and now the last thing I want to do is there's another tool in K exit tools which is called vm core damask which basically takes a dump file and extracts from the dump file the kernel and the kernel lock so this can be used to have a quick look what actually happened and we can that do this here and then you can see okay it crashed it crisp because we sent this is the cue to get a crash okay so well our capture system is working so we're really happy and I edit some
references here for you in case that you want to read a bit more about it this is the source code of the KX ik tools so Nicole get kernel.org
there are some Suzy documentation of how to set things up this is a source code of the kdump program used by Sousa to
make things easier and there's also a nice blocker that explains more or less the same that I explained here so if you don't understand what the ouvea if it wasn't understandable what I explained oh you don't trust me then you can read it there and okay maybe quick some takeaways I think yeah one time oh not too much okay so basically and basically how this works likes it key points that you should recognize or remember you have a reserved memory area and the capture system gets saved in this area in the fqo header is used in points to all these memory locations in your production system where the information can be found in the king capture system then out of this information creates a disk a ton file okay thanks a lot and if
you have any questions please feel free [Applause]
is the capture colonel built cept like with any changes from the production colonel or can you have one built ahead of time that you just reuse that's compatible with your target yeah well it depends so I mean know what we had at Sousa openSUSE do we reuse the kernel because we have the kernel with like everything both billed as a module and then we just put in the energy the modules that we need so you did its that means it's not something you need to reboot the kernel you can if you want to you you could also like built in the the drivers that you need in the kernel and use this but it's not necessary so no okay no more questions so thank you yeah thank you man to you [Laughter] [Applause]