Instrumenting CPython with eBPF
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 141 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 4.0 International: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben. | |
| Identifikatoren | 10.5446/68687 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
EuroPython 202386 / 141
8
17
22
26
27
31
42
48
52
55
56
59
64
66
67
72
73
77
79
83
86
87
95
99
103
105
113
114
115
118
119
123
129
131
135
139
140
141
00:00
Smith-DiagrammTwitter <Softwareplattform>E-MailOSS <Rechnernetz>Twitter <Softwareplattform>SoftwareentwicklerCoxeter-GruppeVideokonferenzProgrammierungE-MailComputeranimationVorlesung/KonferenzBesprechung/Interview
00:35
RechnernetzProgrammDigitalfilterSystemprogrammierungGeradeKernel <Informatik>Elektronische PublikationQuellcodeStatechartEchtzeitsystemFunktionale ProgrammierungDatensichtgerätBinärcodeVirtuelle MaschineGarbentheorieUmsetzung <Informatik>ZahlenbereichObjektorientierte ProgrammierungModulare ProgrammierungGeradeSpeicherbereinigungSystemprogrammierungDynamisches SystemParametersystemBildschirmmaskeDreiecksfreier GraphProgrammanalyseFormale SpracheSoftwareSoftwareentwicklungProfil <Aerodynamik>InterpretiererInformationGenerator <Informatik>LastKonfiguration <Informatik>SystemaufrufHalbleiterspeicherMultiplikationsoperatorMinkowski-MetrikFrequenzKomplex <Algebra>Wurzel <Mathematik>Kartesische KoordinatenFramework <Informatik>PunktZweiEreignishorizontSchnittmengeMaßerweiterungDateiformatFehlermeldungOrdnung <Mathematik>AdditionProdukt <Mathematik>ImplementierungProgrammbibliothekBildschirmfensterProgrammierungWeb SiteFormation <Mathematik>DatenverwaltungComputeranimation
07:27
ProgrammCompilerJust-in-Time-CompilerLesen <Datenverarbeitung>SystemaufrufCompilerSoftwareentwicklungByte-CodeAdressraumProgrammverifikationFunktionale ProgrammierungZahlenbereichMapping <Computergraphik>Prozess <Informatik>ParserVariableAblaufverfolgungFramework <Informatik>ParametersystemEinfacher RingHochdruckHash-AlgorithmusGeradeMechanismus-Design-TheoriePuffer <Netzplantechnik>Minkowski-MetrikGemeinsamer SpeicherHauptidealringZeiger <Informatik>KonditionszahlArkusfunktionLoopZweiMaschinenspracheSchnelltasteProgrammierungFunktion <Mathematik>AusnahmebehandlungSoftwaretestFront-End <Software>Kernel <Informatik>Kartesische KoordinatenPersönliche IdentifikationsnummerCodierungKontextbezogenes SystemSkriptspracheUnendlichkeitFormale SpracheEreignishorizontObjektorientierte ProgrammierungProgrammbibliothekPunktFormation <Mathematik>Computeranimation
14:19
ZeichenketteModulare ProgrammierungBenutzerprofilParserUnicodeStandardabweichungSyntaktische AnalyseObjektorientierte ProgrammierungKette <Mathematik>Kartesische KoordinatenGraphfärbungRechteckKeller <Informatik>MultiplikationsoperatorProfil <Aerodynamik>Abstrakter SyntaxbaumSystemaufrufSoftwareentwicklungHalbleiterspeicherObjektorientierte ProgrammierungMaschinenspracheStichprobenumfangMultiplikationVisualisierungGeradeProzess <Informatik>BefehlsprozessorCliquenweitePhysikalische TheorieHardwareProgrammierungCodeParserSoftwareSystemprogrammierungKernel <Informatik>Elektronische PublikationFunktionale ProgrammierungArithmetisches MittelZahlenbereichSyntaktische AnalyseVererbungshierarchieOverhead <Kommunikationstechnik>GraphRahmenproblemKontextbezogenes SystemFunktion <Mathematik>MultigraphMinkowski-MetrikJust-in-Time-CompilerLesen <Datenverarbeitung>AggregatzustandInterpretiererThreadAbstraktionsebeneTopologieMereologieFormation <Mathematik>Notepad-ComputerGüte der AnpassungInverter <Schaltung>Innerer PunktComputeranimation
21:11
Vorlesung/Konferenz
21:30
Computeranimation
Transkript: Englisch(automatisch erzeugt)
00:04
Hello, everyone. Welcome to my presentation. My name is Furkanandash. I am working as a Python developer at Pampers. In this talk, I will show you how you can instrument the C Python programs using eBPF.
00:20
Before we begin, feel free to ask me any questions via Twitter or email. You can find my presentation in my last video. So what is eBPF? Let's start with the acronym. eBPF stands for Extended Berkeley Package Filter.
00:42
From that name, you can see that its roots lay in network packages. In 1992, Stephen McCain and Juan Jacobson wrote a paper called Berkeley Package Filter. This paper discusses a virtual machine that can run filters, which are programs written
01:01
to determine whether to accept or reject a network packet. These programs were written in the BPF instruction set, a general set of 30-bit instruction that closely resembles the assembled language. On the slide, you can see an example of a eBPF program
01:21
from that paper. In 2014, Alexei Stravoytov and Daniel Bruckmann started work on extended BPF. That is how extended BPF was born. Today, as the official website shows, it's no longer an acronym for anything
01:41
but a standalone name. This is because its capabilities span well beyond the package filtering that its previous name implies. eBPF is an internal machine used to extend the capabilities of the operating system kernel by loading and running programs that don't modify the kernel source code
02:03
or add additional modules. eBPF was originally developed for Linux, but it now also has implementation of Windows. For more information about eBPF, you can visit the eBPF.io website. To use eBPF with CPython,
02:22
we need to make some preparations. Let's start. CPython can be built with the embedded markers, also known as props. These are the dtrace props, which are not activated by default,
02:40
so they have no impact on performance. So what is the dtrace and the props? Dtrace is a comprehensive dynamic tracing framework originally created by some microsystem for troubleshooting the kernel and application problems on the production systems in real time.
03:00
Dtrace framework provides instrumentation points that are called prop. Users can use a prop to record and display relevant information about the kernel or user space process, user process. The prop activation is referred to as firing.
03:21
To use the props, you need to have Python 3.6 or higher. Mac OS comes with the built-in support for dtrace. As a Linux user, I am building the CPython with the embedded props for system tab. System tab provides an infrastructure to simplify the gathering of information
03:43
about running Linux system. It provides a scripting language for instrumentation of kernel or user space applications. Finally, CPython must be configured with the dtrace option. Okay, let's display the available props.
04:02
On Linux, you can verify if the props are present in the built-in binary by seeing if it contains the .not step std section. By executing redelf dash and python command, you can see the .not step std section. In here, we are using the redelf program.
04:24
It's a program for displaying various information about object files on Linux system. On the slide, you can see the prop names and the .not step std section. So let's take a deeper look
04:40
at the available props in the CPython. Function entry indicates that the execution of Python function has begun. Using this prop, you can get the file name, function name, and the line number. Function return, the converse of function entry,
05:03
and indicates that execution of a Python function has ended. Line indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. GEC start fires when the Python interpreter
05:24
starts a garbage collection cycle. First argument is generation to scan, like GEC modules collect function. GEC done, this prop fires when the Python interpreter finish a garbage collection cycle. First argument is the number of collected object.
05:43
Import find load starts. This prop fires before import lib attempts to find and load the module. First argument is the module name. Import find load done fires after the import find and load function is called. First argument is the module name.
06:02
The second argument indicates if the module was successfully loaded. Last prop is audit. Fires when size module's audit function is called. First argument is the event name. The second argument points to a table object.
06:20
I think now that we have a basic idea of EBPF and the CPython props. We can start instrumenting the CPython programs using the EBPF. Before instrumenting, I would like to mention the concept of the instrumentation. In a simple term, instrumentation refers to
06:41
measure of a product performance in order to diagnose errors and to write a transient formation. We can also cover the profiling and tracing in the instrumentation concept. I would like to briefly explain these two concepts.
07:00
In the software engineering, tracing involves a specialized use of logging to record information about a program's execution. Profiling is a form of dynamic program analyzers that measures, for example, memory or time complexity of a program or the frequency and the duration of the function calls.
07:23
I think we are ready to coding. Before doing that, I would like to introduce the BCC. I will write a simple tracing tool using the EBPF. To do this, I will use the BCC Python framework.
07:42
Which makes it very easy to write EBPF programs. BCC, BPF compiler collection, is a toolkit for creating efficient kernel tracing and manipulation programs built upon EBPF. BCC provides kernel instrumentation in C
08:01
and allows the front-end user space application being written in Python or La. Let's start the coding. Thank you. To start with, we are creating an argument parser that will get the process ID. We are adding the PID as an argument.
08:23
So finally, arguments are being parsed. You are seeing an EBPF program. The program consists of a single function, called a trace line, which takes a pointer to struct PTRX as an argument. Inside the program,
08:41
we are declaring the function name. After that, we are using the BPF used read arc function to read address to function name from the second argument. Because the second argument of CPython's lines prop is the function name, we need to get the function name.
09:02
After that, we have BPF prop read user function, we pass the function name from the address to variable function name. Similarly, we are reading the line number from the third argument. Finally, we are printing the function name and the line number using BPF trace print K function.
09:24
In here, we are creating the USD object, specifying the process ID. User land statically defining USDT is a mechanism that allows the BPF program to trace specific events related to the execution of CPython applications.
09:43
After that, we are calling the enable prop function to enable the CPython's line prop, and we are associating it with the eBPF program. In the next line, we are passing the eBPF program as a text, and USD object in the context parameter.
10:06
This line has shared the USD context with eBPF program. And finally, we are creating an infinite loop that continues to run the trace print function. Trace print will print the function,
10:22
this function, print the output whenever a trace event occurs. We keep looping until a keyboard exception. Before running the program, we need a test example program. I wrote a very simple program with for function
10:41
to make it easy to trace. You are seeing the nested calls. This program pins my name, hello Europe Python 23, and its own process ID at one second intervals. On the right side, as you can guess, you are seeing the output of the program.
11:03
Now, let's look at the output of tracing tool. In the first line, we see the process name, Python, and its PID value. Then we see the eBPF trace print k function, and we are seeing the function name as a model,
11:21
because this line is not a function call. For example, this line could be a while loop, or if condition. In our example, it's corresponding to while loop. We can see the function names in the lines just below, because these are the function calls.
11:42
Next to the function names, we are seeing the line numbers. Now we have an eBPF tracing tool that trace line by line the C Python programs. So how does eBPF program work?
12:00
We wrote an eBPF program using the BCC Python framework. eBPF program compiled the eBPF bytecode. After the program's compiled the eBPF bytecode, it's loaded into kernel and verifies. If everything is okay, eBPF bytecode is translated to native code by the JIT compiler
12:23
and executed. eBPF programs are event-driven. It is attached to prop and executed when that prop is triggered. Program brought out data in eBPF maps. The eBPF programs use the maps to collect,
12:41
store, and share data with user space via system call. There are the different of eBPF map, like hash table, array, ring buffers. The tracing tool that we have just written may seem long process for some of us.
13:01
Actually, we could have written this program in a single line. To do this, I would like to introduce you to BPF trace. BPF trace is a high-level tracing language for eBPF. It's language inspired by Eureka and C
13:22
and predecessors tracers such as DTrace and SystemTap. BTrace uses LLVM as a backend to compile scripts to eBPF bytecode and makes use of BCC as a library for interacting with the Linux eBPF system, as well as accessing the Linux tracing capabilities
13:41
and attachment points. Let's look at the one-liner example. I want you to look at the top of the picture. We are seeing a one-liner comment using dash a parameter, we are passing the program.
14:02
Programs begins with the USDD, pet off the CPython and the prop name. It uses the printf function to print function name and the line number. The last thing is the process of ID, process of ID of program, sorry. As you can see, we can easily and quickly trace
14:24
the CPython programs line by line using a one-line program, which is very convenient. When we talk about instrumentation in some Python, our topic may also be about profiling.
14:40
There is a popular tool that call it PySpy. It's a sample profiler for Python programs. It can start profiling any running Python applications just by giving the process ID, which is very easy to use. PySpy works by directly reading the memory
15:02
of the Python program using the process VM read system call on Linux or read call on OSX. Figuring out the call stack of the Python program is done by looking at the global PyInterpreter state variable to get all Python threads
15:22
running in the interpreter, and then iterating over each PyFrame object in each thread to get the call stack. I have an example program written here. This program reads the Python files
15:43
and converts them into abstract tree object. We have a function called file parsers. It will read the Python files and converts them into abstract syntax tree object. In the for loop, in our current pet,
16:04
we are finding the Python files and we are sending this file to a file parser function. Let's profile the example program using the PySpy. Here you see a graph created after profiling by PySpy.
16:24
It demonstrates one of the method of visualization PySpy output. It's called frame graph. Flame graphs are a visualization of theoretical data created to visualize stack trace of profile software
16:40
so that the most frequent code paths to be identified quickly and accurately. Each rectangle in the visualization represent a stack frame. We are seeing the function names, file names, and the line numbers inside the rectangles. The x-axis on the flame chart represents the time
17:01
while the y-axis show the execution stack at that time. The most important things to look at when interpreting flame charts are color and width of the rectangle. Width of each rectangle represents the time spent in that function. So what is the meaning of the color of the rectangles?
17:25
The color can be based on various factors. In here, the red rectangles indicate that they use the CPU more intensively during execution than the orange rectangles.
17:40
I would like to mention another profiler based on eBPF, gProfiler. gProfiler is a system-wide profiler. It combines the multiple sampling profilers to produce unified visualization of what your CPU is spending time on.
18:02
We have already created the flame graph with PySpy. This time, we are looking at a flame graph created with the gProfiler for the same example. We are seeing the different colors. In the top of the picture, we are seeing the Python 3.9,
18:22
which is the Python interpreter. The yellow rectangles are the Python frame. The native code is indicated by the purple rectangles. gProfiler, a system-wide profiler, is a system-wide profiler.
18:42
Unlike PySpy, which operates on per-process-based, it profiles all Python programs that are currently running on the system. If you wish to profile for Python process, you must use the for-distinct PySpy, which brings more overhead. There is no such a requirement in gProfiler.
19:02
With gProfiler, you only need to do this once, and it profiles all Python applications using PyPerf. PyPerf is a part of gProfiler. If you are familiar with the perf tool, you can think of the PyPerf as a kind of super up perf for Python.
19:25
PyPerf is a sampling profiler. gProfiler uses eBPF profiling based on PyPerf. With the PyPerf support, eBPF program can understand C Python stack trace and profile each process without additional overhead.
19:45
So why is the eBPF is important? Performance, eBPF programs have fast access to resource such as memory. For example, they can access the memory of the Python applications much faster than PySpy.
20:03
While PySpy doesn't introduce the overhead on the application itself, but it has some overhead for the system, it needs to access the application memory to extract the stack trace and uses a lot of system calls for that. So this takes time and resource.
20:22
As a result, PySpy can perform efficient and fast profiling, especially in the system wide. No context switches, eBPF programs are executed in the kernel space. As a result, they do not require context switching, so they work faster.
20:40
Event-driven, eBPF programs are event-based, so nothing runs without a specific trigger and you never miss any event. Finally, the kernel just-in-time compiler compiles the eBPF programs into machine code just before the execution, so the code is optimized for the specific hardware
21:02
on which it runs. Thank you for your listening to me. Thank you for the great talk. I was aware of eBPF, but I didn't know you could do that.
21:20
We will not take questions now, but you can find Furkan on the venue and I'm sure he's happy to answer any questions there. Please give it up for Furkan.