We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

lognplot - logging and plotting data from micro's

00:00

Formal Metadata

Title
lognplot - logging and plotting data from micro's
Subtitle
Tracing data on a modern laptop
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Embedded systems are hard to debug. Complex systems have a lot of variables. When debugging those systems, we often log data into some files, and visualize them later on, using excel, matplotlib or something else. This talk is about tracing and logging. What are the options we have as embedded software developers? I will present the lognplot tool, a project to plot incoming data on the fly. Embedded systems are hard to debug. Complex systems have a lot of variables. When debugging those systems, we often log data into some files, and visualize them later on, using excel, matplotlib or something else. This talk is about tracing and logging. What are the options we have as embedded software developers? I will present the lognplot tool, a project to plot incoming data on the fly. There are two implementations, one in python, and one in rust with gtk-rs. The data is stored internally in a zoomable format, allowing large sets of data to be browsed easily. During the talk you will learn how to draw a chart, and how to aggregate large sets of data into summaries. I will demo a STM32 serial wire viewer output connected to this tool to enable live tracing of an embedded system.
LaptopSystem programmingMicrocontrollerDemo (music)Content (media)SoftwareDemonOpen sourceDemo (music)Content (media)Visualization (computer graphics)Presentation of a groupPlotterProgramming languageHoaxBitProjective planeSystem programmingSoftware engineeringComputer animation
DebuggerCodeSystem programmingVariable (mathematics)LogicLetterpress printingType theoryPlot (narrative)BuildingScripting languageData structureLinear mapGraph (mathematics)Vector spaceGraphical user interfaceSource codeSerial portScanning tunneling microscopeFile viewerClosed setMiniDiscCASE <Informatik>BitUtility softwareInsertion lossVideo gameType theoryResultantMultiplication signProjective planeComputer programmingSoftware testingExterior algebraLevel (video gaming)Process (computing)Electronic mailing listWater vaporFeedbackSource codePoint cloudPrototypeVirtual machineUltraviolet photoelectron spectroscopyDifferent (Kate Ryan album)Graph (mathematics)Computer fileGlatte FunktionLaptopPoint (geometry)System programmingScanning tunneling microscopeLibrary (computing)Cartesian coordinate systemAreaMathematicsWeb 2.0CoprocessorPlotterLogicGame controllerWave functionVariable (mathematics)Control flowStructural loadComputer animation
Packet Loss ConcealmentSystem programmingPlot (narrative)BitComputer programmingMicrocontrollerSystem programmingConnected spacePlotterLevel (video gaming)Computer animation
Scripting languageCASE <Informatik>Arrow of timeMicrocontrollerParallel portPlotterLibrary (computing)Computer animation
Plot (narrative)MiniDiscScripting languageStreaming mediaMicrocontrollerMechanism designInterface (computing)Demo (music)Row (database)MiniDiscConnected spaceScripting languageQuicksortComputer fileComputer animation
Menu (computing)Convex hullMotion blurModal logicFunction (mathematics)Row (database)Demo (music)Structural loadLetterpress printingScripting languageStreaming mediaGraphical user interfaceLaptopComputer animation
Zoom lensMultitier architecture2 (number)Computer animationDiagram
Multitier architectureUser interfaceMIDIComputer programmingSemiconductor memoryBefehlsprozessorNoise (electronics)Computer animationDiagram
Multitier architectureExecution unitTwin primeUsabilityArrow of timeLibrary (computing)Component-based software engineeringDiagramForm (programming)Graph (mathematics)Data loggerServer (computing)DatabaseArrow of timeLoginTime seriesFunction (mathematics)Library (computing)Computer programmingGUI widgetMereologyCartesian coordinate systemPlotterComputer animation
DatabaseSeries (mathematics)GUI widgetServer (computing)Client (computing)Insertion lossPoint (geometry)Zoom lensCommunications protocolComponent-based software engineeringLibrary (computing)Network topologyData structureStapeldateiLocal GroupSequenceCalculationLatent heatDatabaseMetadataSemiconductor memoryTime seriesStapeldateiNumbering schemeSampling (statistics)Group actionQuicksortMereologyBitCategory of beingFlow separationComputer animation
StapeldateiSample (statistics)Standard deviationNetwork topologySampling (statistics)Set (mathematics)Maxima and minimaMetadataTrailStapeldateiStandard deviationComputer animation
Sampling (statistics)Network topologyRootkitStapeldateiData structureMetadataComputer animationDiagram
Visualization (computer graphics)Electronic visual displayAlgorithmCurveCartesian coordinate systemLogicCalculationLine (geometry)Graph drawingNetwork topologyPlotterLevel (video gaming)Zoom lensRaw image formatCurveComputer animation
Standard deviationElectronic visual displayArithmetic meanStandard deviationNetwork topologyPlotterMaxima and minimaForm (programming)Line (geometry)Data structureComputer animation
Electronic visual displayPoint (geometry)Graph (mathematics)Level (video gaming)Point (geometry)MereologyFunction (mathematics)Zoom lensComputer animationDiagram
Stability theoryPrototypeGraphical user interfaceServer (computing)Revision controlConnected spaceComputer programmingImplementationWrapper (data mining)SoftwareMachine codeDemo (music)Library (computing)CodeComputer animation
Limit (category theory)Multitier architectureMultiplication signArtistic renderingQuicksortPlotterGraph (mathematics)Computer programmingDemo (music)Computer animationDiagram
Execution unitFibonacci numberLimit (category theory)Convex hullInfinityMenu (computing)Demo (music)Scripting languageComputer fileStapeldateiRow (database)Sampling (statistics)DatabaseMultiplication signComputer animation
Zoom lensMultitier architectureConvex hullInclusion mapInterior (topology)Execution unitInflection pointPoint (geometry)Tracing (software)Computer animationDiagram
Multitier architectureOnline helpExecution unitGamma functionConvex hullMultiplicationDrag (physics)Computer programmingTracing (software)Revision controlComputer animationDiagram
View (database)PlotterComputer animation
Menu (computing)Core dumpRevision controlDemo (music)Computer programmingScripting languageFlow separationComputer animation
Video gameGraph coloringZoom lensSemiconductor memoryRevision controlComputer animationDiagram
Execution unitWechselseitige InformationDemo (music)WhiteboardArmMultiplication signComputer animation
Plot (narrative)Game controllerLink (knot theory)Software testingClient (computing)Link (knot theory)WhiteboardClient (computing)Computer hardwareMicrocontrollerComputer animation
Function (mathematics)ArmExecution unitRead-only memoryTimestampInterface (computing)Library (computing)Graphical user interfaceData streamConfiguration spaceReading (process)Address spaceStrategy gameStrategy gameDialectSemiconductor memoryMereologyEvent horizonExecution unitBlock (periodic table)Computer programmingTimestampTracing (software)System programmingArmComputer animation
Sample (statistics)CodeWhiteboardLink (knot theory)DisintegrationWhiteboardSemiconductor memorySampling (statistics)Computer programmingLoop (music)Computer animation
Chemical equationTime zoneWechselseitige InformationDuality (mathematics)Convex hulloutputServer (computing)Computer animation
Zoom lensMultitier architectureAsynchronous Transfer ModeWhiteboardTracing (software)Execution unitMultiplication signComputer animationDiagram
Demo (music)Source codeLink (knot theory)FeedbackComputer animation
Open sourcePoint cloudFacebookOpen set
Transcript: English(auto-generated)
Good evening every everyone Glad to see still some some people around This talk is about a lot and plot tool, which I created
Yeah in the past month So the the contents of this presentation is a little bit who am I? What is this and then I hope to give some demos about? What I did So who am I? Yeah, I'm currently working at them collision as a software engineer
I've used programming languages Python C and rust and I've also worked on other projects like a PPI and risk Python and This presentation I will present about a lot and plot
As I think it's a good So what is this talk? This will be a little bit of a weird stock It's a bit fake But what I wanted to make was a tool which which you can visualize data from from systems
And this in a with live data and also zoomable So let me first start with a little bit of motivation. Why did I do this as mentioned this morning? almost all talks here the motivation is because Because you can and for me that's also the case it's enough but there's more motivation here
So there's when you debug systems. There's basically Yeah, two types of debugging you have either you you use a GDB and you start your program and you
Stop and step your program inspect variables Make variable watches and it's all very great and the tooling is also very good, but this does not always work If you have a controller for for example a motion stage you don't want to stop the The program in the middle because the stage will leave the leave the room which is not good
So we cannot break point the program So what do we do well, for example, you could do print F or you can toggle an LED Or you can monitor variable changes and I think we we can do better in this area
So what what is the system tracing involved? It's it's really inspecting the System in a test we built a whole machine and to do a test with it and typical With these kind of set ups is that
Testing is expensive time is limited. You have a lab. You're not the only one who has the lab A test takes time. So maybe it takes one hour. So in each day you only have eight attempts Well, that's not much Also resource are limited electricity water whatever you may use it's not infinite so you cannot
Repeat your test indefinitely Another problem with system testing is that tests are often not? As repeatable as you want them to be where as soon as you test you run the test once you run it again exactly the same
With the motion stage you run it once you run it again different result every time so one solution to this is Simply loss everything and analyze later you do the test once and you can you you have all the data that you that you need Some of my own experiences here with the
tracing of data I've I built in the last years several tracing utilities in Matlab in C sharp manuscripts in Python with different libraries and
I was not Yeah, really satisfied with this result Most of these libraries are intended for creating smooth graphs for publications, which is then not handy to use in a refreshing way or in a in a life data way
Well, maybe there's already an existing project. There are many projects. I have here a few but there are many more and After this talk, I hope to get Feedback from you The project I missed I want to make a list in the readme which lists many alternatives to to plot and
Visualize data from a life system. So KST plots from KDE. It's very good. It's very mature you can Load very big CSV files, but still it works on the CSV file. So you first have to create the CSV file There's another project plot juggler
Look looks promising. I have to try it out more Say rock is really cool user interface, but it's really focused I think on Getting if getting data from logic analyzers, there's even more tools GTK wave
But it's more for Digital logic so in that area then there's of course Ross operating system, which are 2d Also plotting tool, but then you're tied to Ross There's also a tool STM studio
Which allows you to monitor an STM processor And then there's of course whole slew of Web based tools, but I want to run it on desktop So that's a little bit of motivation. There's a lot but I Want something else? So, what do I actually want I
Want first of all, I want a desktop application. I want to do this offline in the lab And I don't want to use some clouds thing Of course having your data in the cloud is good because then you can share your data But for now, I want to keep it on my laptop. I Want to be able to zoom and pan and append data also log it on a file
preferably not CSV, but Yeah, probably the first prototype will be CSV and I want to get data from many different sources So usually you have a system with many different Processes in it. Well now go into the Program that I created for to do this a little bit about the design and how it's made
So top level is really simple ID We have a plot tool and we have a TCP connection and we have various systems we have a PC and we have a microcontroller and we send the data from the microcontroller and
the PLC system to the to the lot to the plot tool Okay So example use case yeah something like this We have a microcontroller and we have a Python script interfacing with this microcontroller and the Python script will center data over TCP
And next to that running in parallel we have an oscilloscope and which is also interfaced with for example the fisheye bush or Yeah, some Python library which allows you to communicate with the oscilloscope and this will also send the data to the plot tool One note is that the data is the data arrow is one-directional
And this allows for some interesting stuff for example We can we can Make despite the script interface it with a microcontroller and stream the same data to a disk and then later on We can load again this file from disk and again feed it via a TCP connection into the plotting tool
so then you have a sort of a Recording mechanism of what happens if you do this with all your data, you can replay it. So now for the first demo
So this demo I Created a GUI using rust and GTK which exactly this concept
this is a demo of having a Record which is a tool which loads with opens the microphone of this laptop and streams its to Standard output and then I pipe it into a script which will take this binary format and send it via TCP into the plotting tool
When I start this yeah, I have some prints of the data and In this this is the The rest tool and I can drag this data here and you see the data
Flowing in I can also zoom to the last 30 seconds or last 10 seconds or even Last seconds, which is nice If you do this Can make some noise maybe Some data, okay, which is good. I can also then you can pen this data and you can zoom this data
By scrolling into it. So this is handy. So this is good and we can also check the CPU usage of this program It's here. So we see that even if it's while it's ingesting a
Fair amount of data the usage is not that that large that's but You see the memory increasing but yeah, that's what you expect. It's audio data. It's not compressed with whatever
I will stop this for now. I will not keep it running but So what is the design of this log tool Primary idea is to have a reusable library
Which can be included in other applications for to have this plotting functionality and a showcase tool So what you just saw was the showcase tool it showcases. This is a nice graph and it works Yeah This is what it can do. So the graph of the program you can include in your own application. That's the design
so as you can see here, there is the Log and plot tool which will use the library and the library will have several things widgets which display the graph a charting Function or charting functions, which draw the graph
then a TSDB which stands for time series database, which will Have to the data in such a form that you can easily zoom and pan it and also a TCP server Which can receive the data notes that the arrows indicate the?
Dependency so you can pick for example the TCP server in the time series database and use only that part of the library So as several parts of this log and plot library, I will now mainly zoom into the time series database
How that is how that works? A side note time series database is very interesting. It's a whole new scheme of databases Specific for time series. So this is what I made was not really a time series database as such but ideas are similar
some Properties of the time series database. I made some assumptions data can only be appended and
So I will now go into a little bit of how this signal data is structured in memory so what I did was create a sort of a b3 and And group the samples in batches and keep metadata for each batch. So what does this mean? It means we get the data in as separate samples. We split it into batches of a certain amount of samples
For each batch we have a Set of metadata such as when was the first sample received one was the last sample received. What was the minimum value? What was the maximum value and what was the mean? What was the standard deviation these kind of things we keep track of that for each batch
Next we organize these batches into a b3 Which is a tree structure with a root node intermediate node and leaf nodes. So the samples will be in the leaf nodes and each leaf node will have
Metadata about this sample about the samples, but the intermediate nodes will also have this metadata So now what can we do when we visualize this data? So when we are really zoomed out We will only
Go into the tree to a certain level we will not go all the way to the Raw samples, but if you're really zoomed out, we will only go to halfway the tree so when we draw the
The plots we draw the access and we draw the curves depending on our zoom levels and we draw legends So what does this mean in zoomed out form when you zoom out into the plots you will see the min max mean and standard deviations Like is displayed here. So the you see the mean line in the middle and then the
transparent Region indicates the min max of this signal and if you remember the tree structure The previous graph was made From intermediate level nodes, so not really zoomed into the leaf nodes
When we zoom in you see individual data points and when you see something like this Then we're really at leaf nodes, but only at the specific part you zoomed in of the data So this is in a nutshell how this zooming functionality works so As I already said I demoed the first demo with the Rust implementation
So there's actually two implementations one in Python and one in Rust. So why did I did it like that? Python is handy. If you want to try out a new ID Python is ideal because it works really fast But the performance is maybe not that good. So that's why I also made a Rust version which compiles really to machine code
Unfortunately, the Rust version is made on GTK and the Python on PyQt, but it's good to have two redundant implementations to Try out things In the Rust version I used Tokyo RS Crate that is a
TCP or a Networking crate to handle TCP connections and also made a C wrapper around the Rust library So you can use this from C code you can link to this library to for example
Send a data from your program. So I will now demo other stuff See so we already saw the audio signal, which is nice but What I also
quickly added to this program was a sort of Was the the rendering time of the graph itself as a signal because we can render signals So in this graph, you will see the the rendering time of this plot. So the the rendering time is always around
four milliseconds so yeah, you can also zoom in on this an other demo that I Have is This is a batch download demo. So say you have made a recording and you want to just look into this data
See what happened You can make a Python script we will which will load in the SQLite database CSV file txt file whatever you have and send it to this tool and this script will send 10 million samples to the plotting tool
So it will it will take some time But not not that long So we can delete it and again we can zoom this data. So these are 10 million data points and again, you can zoom it as before and this is another name of it generates some some noise and
It will send to two traces drag tutor to traces you can also
Drag multiple traces in the in the program So this is the rest version I can also demo the The Python version So this is the the the Python with pikute version if you prefer to use Python you can use this one
you can switch here to You can switch to multiple views. I made this view that you can have multiple
plots But I will now use one plot and you can also view it here And I can do that I can start the same demo as before and send the data not to the rust Program, but to the Python program and usually you should also benefit of having the scripts which interact with your hardware separate from the
GUI to visualize it because now I have the same signals But in this in this Python version and as you can see I implemented in the Python version already something Extras, which is that this color indicates that these signals are sort of life. So the the data was arrived
And recent And you can also drag them here and zoom in So you can see that the Python version version actually also performs pretty reasonable pretty good But the memory consumption is rather rather large
Yes, I wanted to give an arm demo, but I'm not sure
If there's enough time for it But let me explain the ID so I made a I made an another example which uses this board It's microcontroller board with an STM 32 on it just to really get hardware and lock this data into the to the plotting tool
The setup is like this We have a microcontroller and an ST link which is on this board and then I made a rust client which interacts with this ST link and send this data also to this plotting utility and I wanted to highlight this this is made by made available by Armin is
present in many Or arm course. It's the traceport units the data watch point and trace and an instrumentation trace Blocks which allow you to do this tracing kind of thing
So not to stop and continue debugging but you can configure these devices to Monitor memory regions and whenever this memory region changes it will generate an event with the program counter so the The part of your program which changed the memory and the memory value and a timestamp So this is ideal for tracing and not holding the system. So the tracing strategy here is to
Use the staling and Connect to this as tailing using the art our USB library, it's a rust crate to interface with USB and
Send this data again to the tool This is the sample program I made really small loop with just one volatile variable so that I have a memory access which is changed So you have to believe me that is in this board. I will now check if this works. I have to
Make sure that I have to start the plotting tool first because that will open a TCP server. Yes
So we see here the the signal which is coming from this board
so it will send the data and this board is now running in free running modes and we capture each time this variable a is written and with the data watchpoint and trace unit and I Gather this event and send it to the to the plotting tool. That's all
Thank you for your attention. The link is here. So all kind of feedback is welcome. If you want to add something Please do so
Thank you