Monitoring Kubernetes and Virtualization

Video in TIB AV-Portal: Monitoring Kubernetes and Virtualization

Formal Metadata

Monitoring Kubernetes and Virtualization
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Now that Kubevirt, a converged infrastructure that brings virtualization on top of Kubernetes came into the world, it’s time to take a look on the monitoring aspects of this solution. This talk will go through some of the popular monitoring solutions and integrations that exist for Kubernetes, a sneak peak of how monitoring is done for Virtual machines on oVirt's open-source distributed virtualization solution and how could it be applied on Kubevirt. Attendees with Kubernetes experience could learn more on monitoring solutions for both Kubernetes and virtualization.
Software Virtual machine Virtualization
Metric system Key (cryptography) Connectivity (graph theory) Source code State of matter Virtual machine Data storage device Virtualization Database Mereology Computer Component-based software engineering Software Mixed reality Matrix (mathematics) Computing platform Metric system Computing platform Physical system
Server (computing) Metric system Open source View (database) Connectivity (graph theory) File format Time series Primitive (album) Client (computing) Thomas Kuhn Query language Visualization (computer graphics) Matrix (mathematics) Vertex (graph theory) Endliche Modelltheorie Series (mathematics) Physical system Source code Focus (optics) Key (cryptography) Block (periodic table) Server (computing) State of matter Client (computing) Component-based software engineering Data management Query language Order (biology) Point cloud Physical system
Metric system Run time (program lifecycle phase) Structural load State of matter Multiplication sign View (database) Source code Primitive (album) Mereology Total S.A. Data transmission Dimensional analysis Bit rate Semiconductor memory Befehlsprozessor Core dump Matrix (mathematics) Process (computing) Bit Representational state transfer Statistics Type theory Befehlsprozessor Process (computing) Queue (abstract data type) Metric system Statistics Game controller Server (computing) Service (economics) Connectivity (graph theory) Average Read-only memory Queue (abstract data type) Speicherbereinigung MiniDisc Installable File System Computer architecture Game controller Server (computing) Debugger Computer network Core dump Limit (category theory) Cartesian coordinate system System call Component-based software engineering Cache (computing) Query language Cube Object (grammar) Scheduling (computing)
Scheduling (computing) Game controller Statistics Metric system State of matter Connectivity (graph theory) Channel capacity Source code Limit (category theory) Number Read-only memory Term (mathematics) Semiconductor memory Phase transition Set (mathematics) Matrix (mathematics) Boundary value problem Utility software Process (computing) Vertex (graph theory) Sanitary sewer Game controller Demon Server (computing) State of matter Counting Instance (computer science) Limit (category theory) Statistics Component-based software engineering Befehlsprozessor Query language Cube Phase transition Queue (abstract data type) Metric system Scheduling (computing)
NP-hard Mainframe computer Data management Virtual reality Mixed reality Multiplication sign Computer hardware Virtual machine Allegory Utility software Computing platform
Workload Disintegration Data storage device Virtual machine Code Virtualization Physical system Modem Point cloud
Game controller Server (computing) Game controller Server (computing) Connectivity (graph theory) Virtual machine Virtualization Virtualization Open set Type theory Video game Mixed reality Cycle (graph theory) Scheduling (computing)
Reading (process) Statistics Scheduling (computing) Metric system Multiplication sign Flash memory Virtual machine Total S.A. Writing Latent heat Virtual reality Read-only memory Semiconductor memory Befehlsprozessor Matrix (mathematics) MiniDisc Plug-in (computing) Source code Operations research Theory of relativity Information Memory management Computer network Database Virtualization Statistics Virtual machine Transmitter Befehlsprozessor Error message Software MiniDisc Library (computing)
Component-based software engineering Statistics Metric system Server (computing) Connectivity (graph theory) Cube State of matter Virtual machine Matrix (mathematics) Vertex (graph theory) Statistics
Mobile app Server (computing) Game controller Functional (mathematics) Metric system Service (economics) Connectivity (graph theory) Virtual machine Primitive (album) Client (computing) Field (computer science) Operator (mathematics) Matrix (mathematics) Videoconferencing Vertex (graph theory) Series (mathematics) MiniDisc Library (computing) Dependent and independent variables Exponentiation State of matter Client (computing) Virtualization Component-based software engineering Digital photography Data management Personal digital assistant Order (biology) Object (grammar) Metric system Library (computing)
Game controller Game controller Scheduling (computing) Statistics Server (computing) Metric system Server (computing) Connectivity (graph theory) Multiplication sign Representational state transfer Statistics Number Cache (computing) Cache (computing) Mixed reality Queue (abstract data type) Series (mathematics) Metric system Scheduling (computing)
State observer Statistics Greatest element Metric system Real number Connectivity (graph theory) Multiplication sign Virtual machine Time series Open set Mereology Coprocessor Virtual reality Read-only memory Semiconductor memory Matrix (mathematics) Utility software Process (computing) Ecoinformatics Form (programming) Information File format Software developer Virtual machine Type theory Data management Process (computing) Befehlsprozessor Software Metric system Virtual reality
Metric system Graph (mathematics) Connectivity (graph theory) Virtual machine Primitive (album) Virtualization Complete metric space Cartesian coordinate system Complete metric space Discrete element method Virtual reality Befehlsprozessor Qubit Semiconductor memory Mixed reality Software framework Extension (kinesiology) Computing platform Extension (kinesiology)
so hi this talk is going to be about
monitoring converge infrastructure containing and running together containers and virtual machines together the main course here will be about metric collection because I can't cover all of it so let's start my name is the Anil Quinn
I'm working in red at CES 2016 and I'm a part of their kubera team so we'll
briefly talk about kubernetes and metrics collection in kubernetes the main sources the key sources for metrics collection we'll see how it comes together for Matthias to provide us a full monitoring pipeline solution having covered that we want to throw in virtual machines into the mix so we'll briefly look what Qbert is that brings virtual machines on top of kubernetes see from past experience what we have collected the for virtual machine and matrixes and see how it all comes together eventually so we have kubernetes as a start as our container centric platform Orchestrator that orchestrates computational nodes and pods network storage database all of components that you would like to monitor in your system and the second
building block here is prometheus Prometheus was is was originated in Sound Cloud in 2012 now it's an open source community it's a monitoring and alerting system it's a multi-dimensional label based model it based it's based on a time series the data which is comprised of a key key value pairs it has a very strong a query language promiscuous prompt QL that analyzes that time series data and you can see it really integrates in a natively almost way with kubernetes Sam the community's components already has integrated or have integrated the primitives a quick view on the model so in a minute
we have the Prometheus server that scrapes and pooled that through HTTP endpoints into its server collects then aggregates that data on the data you can perform a prompt QL series in order to analyze it and move it on aggregate the data and even create the alerting alerting are pushed to the alert manager into its clients and you can also use grow fauna or other API clients to visualize that data but again our main focus here is matrix collection so because Prometheus needs as much as auto discovery as it can get it needs to scrape data pull data from matrix endpoints HTTP endpoints for that it's very easy to integrate with kubernetes since it exposes these endpoints on on API is on HTTP as I said some of the matrix is already exposed through the matrix endpoint and some we need to be added and collected so I'll go over two
or three sources of matrixes in kubernetes in the community say architecture the first one of them is
host matrix kubernetes notes so kubernetes knows didn't have natively exposing matrix endpoints for that we they had to write an exporter an exporter is Prometheus exporter is a service that collects you all of your data from your node for that good example or any other data you need to expose from your application etc exposes them on the permittee assay endpoint so primitives can come discover it and scrape them out so node node metrics are scraped from proc and you get all your familiar house statistics such as node CPU node CPU for example has like 15 dimensions of matrix type Rossi to you like either I await but if you want to analyze for example how much time the CPU spent on a node or pair CPU core you can perform a promise QL and remove these and get your note CPU statistics are there are of course memory memory available you can also perform queries against the total memory left there are more advanced matrix then only note statistic if you look at it in a kubernetes kind of way for that we have another component called cube state metrics which looks at it from the kubernetes way and also takes into account memory limits and CPU limits moving on so in our house we have containers and containers metrics are exposed to see advisor see see advisor provides you data about containers runtimes such as docker or cryo it is embedded in the cubelet so cubelet natively exposes an endpoint to two slash metrics which is a port for for Prometheus so we don't need to add additional exporters to collect that data so you analyze these metrics and resources usage and expose them on the metrics these are also also call the core metrics for each container and again much like know just in just in a container view you have your original CPU matrix in memory memory actually this matrix is very advanced it's it's a in the nimac so you know it gives you actually the exact memory usage you have left taking into consideration all other matrix you have and you have under your container and of course memory transmission another part let's look a bit on the orchestration parts so I'll talk about only API Silverhill so because the API server is the front end for kubernetes cluster and gets all the calls and relays them to etcd in controller it's an important component to monitor for example you want to create a pod or run a deployment you want to see and it's all done through REST API calls so you want to see you want to see the request rates and latency how long did it take for me to complete a request to create such a deployment for example or how long did it take even counting the amount of that and let's say you want to create a thousand deployments on your cluster when you create a deployment it goes on to work use for controllers it accumulates in cache the the request for creating objects or deployments and if your queue gets accumulated and get stuck it's it's you want to detect that bottleneck so it's a very important aspect to monitor because kubernetes is based on goal so you get out-of-the-box goal and statistics garbage collection memories Fez etc and you also monitored the service process itself so that was only one component of
course you have metrics collection for scheduler the kubernetes controller etcd but I won't get into those now there's a
an add-on called cube state matrix this add-on you need to deploy as an instance only one instance in your kubernetes cluster and it's more its relates more to not health or utilization it focuses on counts and status so for example you want to check what what status is your pod boding in terms of scheduling or health not helping the phase it is in you have the concept of CPU limits or memory limits we won't allow a pod to run containers without any boundaries to exhaust all of them so I want to check statistics how how much memory you have in according to your limits or before or provide even more advanced their queries on these statistics there are a thousand of examples in cube state metrics I'll talk about for example the replica sets you have defined a replica set replica set you want to run a number of deployments on your cluster and you want to make sure that these number of deployments I deployments are running in Trussell so you want to see if they are not available how many are there in the cluster and so on so that was very briefly about kubernetes components and sources for metrics collection and now
we want to add virtual machines into the mix so Qbert Qbert comes to close the gap between virtual machines and containers under the same under the same hood and it converts infrastructure we want to run also virtual machines also containers in our in the same platform and we ask yourself why so this is a
great allegory containers are considered to be lightweight they're fast to bring up their natively they have native performance but virtual machines with their bulky heavy utility hard Hardware belt are not going to disappear we'll have them in the next five or ten years at at least and we'll still want to maintain them you can look at an example for mainframes it's taken a long time to face house mainframes they're see they still even exist so so what can I do with first of
all we want to bring the legacy we want to run it under the same folder we don't want to maintain two infrastructures we don't want to maintain two infrastructure so we're going to mix and match and add virtualization on top of kubernetes that's also getting all the kubernetes advantages such as scheduling the storage networking well some and fixes to manage to do that we are also in the use of Qbert and running virtual machines on the kubernetes cluster you can also enable and run any of us who want to run on coop on top of kubernetes so this is governor this and we added to
the mix some virtual machine components again I won't go too deep into the components so on the left you can see the node in the node we have a pod which runs a vm via virtual machines runs on pods you know and you have also side the containers on that pod to enable utilization such as they lived it they also have key components very resembling kubernetes components like via type i same as api server to create your virtual machine CL these pass them on to you TC d having the built controller identifying that creating a pod running a virtual machine on a pod and also scheduling it to a node once it's scheduled to a node the virtual machine gets the nodes assigned and then all the handling of the vm life cycles moves to the virtual handler again on top of the the fork so of course I added something
new I have monitoring on kubernetes I want to also monitor virtual machines so I want to figure out first what to collect so a quick example from pure virtualization overt other it uses
collect D to collected matrixes it has powerful plugins a lot of plugins they can add 2 to it and run it it it gets you how statistics database statistics angel engine stats and more more important for us right now it's virtual machine statistics so for virtual
machine statistics we have a wheeled plugin will again communicates with library and library gives us the information about the virtual machine so of course you are introducing here some new aspects not necessarily all will be monitored and adapted to kubernetes but virtual CPU how much time I get was running on a CPU memory some specific related memory matrix like ballooning or over-commitment memory management matrixes disk reads and white flashes for discus for disks and also network aspects some of them might merge from the benefit of kubernetes but we might want to adjust them to virtual machines but some will need to add one strong example I can give you that kubernetes has a scheduler a very strong catch schedule but it doesn't have like a tea scheduling policies we have in virtualization and we want and we'll probably add these policy also to the scheduler so we want to see why our cluster behaves in such scheduling matter like let's say hi variable H a reservation and we also want to monitor that maybe we'll decide in the future to change the scheduling policy so it's a very important aspect so we consistently
similarities and add in a virtual machine statistics so coming back again
to prometheus and kubernetes we had our collection node exporters cube components cubed matrix and of course
we're missing the a collection for virtual machine virtual machines and cubed just came in so we need to write a custom exporter very much like not exported so for that we need primitives exporters writing an exponent can be done in Prometheus client library for us it was very easy it's mainly in go so it's very native to write it so that that exporter will collect your data from the app matrix it will expose it on matrix endpoint from kubernetes flash matrix and it will be available to scraped by Prometheus server so we'll add custom X photos for our virtual machines
and we'll now start pulling other components so we have the kubernetes components and now we talked about virtual machine matrix we talked about components for video for orchestrating the virtual machines such as they have built API in the field controller we have also processes that running inside the pod as I said for enabling virtualization such as library and qmo I want to complete the picture in a small note about the whole cluster in order to deploy Prometheus servers on a kubernetes cluster we can use very easily prometheus operator it's from it's it has two main functionalities its first of all to deploy and manage the primitive server you get also alert manager you can get also note exporters or other responses you may define and also important to watch your components on the cluster so we added virtual machine and in order to watch them we have a previous operator we have an entity called a service monitor you define the port you define the objects you want to monitor you add the corresponding labels for that matter and if you now create new series it detects that and start scraping these these metrics also for virtual machines in our case
so cubed metrics so I'm taking you back so we looked at metrics we collect for kubernetes some came from the cubelet some came from node and components it is CD scheduler and controllers and now adding to our mix we add the very controller and we have the very PI and the veiled handler and I and I told you it's very something the kubernetes components so if we look back at the API
server we talked about it square communicating with controller walk use and performance it's it has native statistic about goal and and it also handles the REST API requests and measures them so very similar we have the convert will control a walk use we want to want to go over that and get statistics about that we have cue Layton series work duration and number of the time also the caches that comes from go race REST API calls again very similar to the API so low
and now I want to focus specifically on virtual machine' statistics so the previous were general components in a kubernetes cluster here you drill down into the virtual machine statistics and matrixes so two major parts virtual machine matrix which we saw how how and what we collect in overt in a pure virtualization environment and processes so for processor its I'll say it's more for deployment the developer utility and less for real cluster and management and observability and these processes for example are also scraped from proc like a node exporter and we get metrics about them okay so some some important there
information about being metrics we can see the four pillars of metrics we collected in Oviatt and usually these pillars are repeating themselves in all of the matrix these are not the only matrix will collect but these these are the most prominent example so for example V a V n CPU or CPU time I want to see my guest CPU how much time they use memory for example you want to create a matrix that is composed for several matrixes how much how much memory we haven't said how much actual ballooning was left for the VM itself our assess the amount of it all compiling and coming down to to completing a collect matrix and also about network on the on the bottom you can see the pod informatics which I talked about processes themselves and you can see a full form of matrix how it looks in a Prometheus expose format you have the you have the name of the metric himself itself which domain or for our or matter what VM does it monitor you can add a host or other labels for that matter also we added the process name and the type of matrix this applies to all metrics time all metrics time when time series format we want to show here so this is an example
how how it will look on a graph on a dashboard of course you can also use caba?a for that matter or other tools for a very simple example for a VM running on a pod is CPU usage and memory usage okay so in conclusion that was a
very fast talk about some some concepts but the main take here is that we have we have kubernetes and primitives we have this whole platform and framework and solution existing we have very easy extensibility for that so for us by adding a new add-on qubit and virtualization in turn a mix monitoring that was kind of native and very supportive and for us we got a complete monitor instruction and it's continuing to grow as we speak for convergence for social infrastructure but it's it shouldn't be just only for virtual machine everybody can take it for another is is himself and if he adds new components on top of kubernetes on add-ons or side side application we can monitor that using the concepts of primitives exporters and are there many notions


  383 ms - page object


AV-Portal 3.19.2 (70adb5fbc8bbcafb435210ef7d62ffee973cf172)