lognplot - logging and plotting data from micro's
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47272 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
LaptopSystem programmingMicrocontrollerDemo (music)Content (media)SoftwareDemonOpen sourceDemo (music)Content (media)Visualization (computer graphics)Presentation of a groupPlotterProgramming languageHoaxBitProjective planeSystem programmingSoftware engineeringComputer animation
01:29
DebuggerCodeSystem programmingVariable (mathematics)LogicLetterpress printingType theoryPlot (narrative)BuildingScripting languageData structureLinear mapGraph (mathematics)Vector spaceGraphical user interfaceSource codeSerial portScanning tunneling microscopeFile viewerClosed setMiniDiscCASE <Informatik>BitUtility softwareInsertion lossVideo gameType theoryResultantMultiplication signProjective planeComputer programmingSoftware testingExterior algebraLevel (video gaming)Process (computing)Electronic mailing listWater vaporFeedbackSource codePoint cloudPrototypeVirtual machineUltraviolet photoelectron spectroscopyDifferent (Kate Ryan album)Graph (mathematics)Computer fileGlatte FunktionLaptopPoint (geometry)System programmingScanning tunneling microscopeLibrary (computing)Cartesian coordinate systemAreaMathematicsWeb 2.0CoprocessorPlotterLogicGame controllerWave functionVariable (mathematics)Control flowStructural loadComputer animation
06:56
Packet Loss ConcealmentSystem programmingPlot (narrative)BitComputer programmingMicrocontrollerSystem programmingConnected spacePlotterLevel (video gaming)Computer animation
07:27
Scripting languageCASE <Informatik>Arrow of timeMicrocontrollerParallel portPlotterLibrary (computing)Computer animation
08:07
Plot (narrative)MiniDiscScripting languageStreaming mediaMicrocontrollerMechanism designInterface (computing)Demo (music)Row (database)MiniDiscConnected spaceScripting languageQuicksortComputer fileComputer animation
08:37
Menu (computing)Convex hullMotion blurModal logicFunction (mathematics)Row (database)Demo (music)Structural loadLetterpress printingScripting languageStreaming mediaGraphical user interfaceLaptopComputer animation
09:51
Zoom lensMultitier architecture2 (number)Computer animationDiagram
10:12
Multitier architectureUser interfaceMIDIComputer programmingSemiconductor memoryBefehlsprozessorNoise (electronics)Computer animationDiagram
11:02
Multitier architectureExecution unitTwin primeUsabilityArrow of timeLibrary (computing)Component-based software engineeringDiagramForm (programming)Graph (mathematics)Data loggerServer (computing)DatabaseArrow of timeLoginTime seriesFunction (mathematics)Library (computing)Computer programmingGUI widgetMereologyCartesian coordinate systemPlotterComputer animation
12:29
DatabaseSeries (mathematics)GUI widgetServer (computing)Client (computing)Insertion lossPoint (geometry)Zoom lensCommunications protocolComponent-based software engineeringLibrary (computing)Network topologyData structureStapeldateiLocal GroupSequenceCalculationLatent heatDatabaseMetadataSemiconductor memoryTime seriesStapeldateiNumbering schemeSampling (statistics)Group actionQuicksortMereologyBitCategory of beingFlow separationComputer animation
13:44
StapeldateiSample (statistics)Standard deviationNetwork topologySampling (statistics)Set (mathematics)Maxima and minimaMetadataTrailStapeldateiStandard deviationComputer animation
14:04
Sampling (statistics)Network topologyRootkitStapeldateiData structureMetadataComputer animationDiagram
14:32
Visualization (computer graphics)Electronic visual displayAlgorithmCurveCartesian coordinate systemLogicCalculationLine (geometry)Graph drawingNetwork topologyPlotterLevel (video gaming)Zoom lensRaw image formatCurveComputer animation
15:07
Standard deviationElectronic visual displayArithmetic meanStandard deviationNetwork topologyPlotterMaxima and minimaForm (programming)Line (geometry)Data structureComputer animation
15:29
Electronic visual displayPoint (geometry)Graph (mathematics)Level (video gaming)Point (geometry)MereologyFunction (mathematics)Zoom lensComputer animationDiagram
15:58
Stability theoryPrototypeGraphical user interfaceServer (computing)Revision controlConnected spaceComputer programmingImplementationWrapper (data mining)SoftwareMachine codeDemo (music)Library (computing)CodeComputer animation
17:11
Limit (category theory)Multitier architectureMultiplication signArtistic renderingQuicksortPlotterGraph (mathematics)Computer programmingDemo (music)Computer animationDiagram
17:49
Execution unitFibonacci numberLimit (category theory)Convex hullInfinityMenu (computing)Demo (music)Scripting languageComputer fileStapeldateiRow (database)Sampling (statistics)DatabaseMultiplication signComputer animation
18:27
Zoom lensMultitier architectureConvex hullInclusion mapInterior (topology)Execution unitInflection pointPoint (geometry)Tracing (software)Computer animationDiagram
18:52
Multitier architectureOnline helpExecution unitGamma functionConvex hullMultiplicationDrag (physics)Computer programmingTracing (software)Revision controlComputer animationDiagram
19:20
View (database)PlotterComputer animation
19:50
Menu (computing)Core dumpRevision controlDemo (music)Computer programmingScripting languageFlow separationComputer animation
20:11
Video gameGraph coloringZoom lensSemiconductor memoryRevision controlComputer animationDiagram
20:45
Execution unitWechselseitige InformationDemo (music)WhiteboardArmMultiplication signComputer animation
21:05
Plot (narrative)Game controllerLink (knot theory)Software testingClient (computing)Link (knot theory)WhiteboardClient (computing)Computer hardwareMicrocontrollerComputer animation
21:36
Function (mathematics)ArmExecution unitRead-only memoryTimestampInterface (computing)Library (computing)Graphical user interfaceData streamConfiguration spaceReading (process)Address spaceStrategy gameStrategy gameDialectSemiconductor memoryMereologyEvent horizonExecution unitBlock (periodic table)Computer programmingTimestampTracing (software)System programmingArmComputer animation
22:43
Sample (statistics)CodeWhiteboardLink (knot theory)DisintegrationWhiteboardSemiconductor memorySampling (statistics)Computer programmingLoop (music)Computer animation
23:02
Chemical equationTime zoneWechselseitige InformationDuality (mathematics)Convex hulloutputServer (computing)Computer animation
23:35
Zoom lensMultitier architectureAsynchronous Transfer ModeWhiteboardTracing (software)Execution unitMultiplication signComputer animationDiagram
24:04
Demo (music)Source codeLink (knot theory)FeedbackComputer animation
24:30
Open sourcePoint cloudFacebookOpen set
Transcript: English(auto-generated)
00:05
Good evening every everyone Glad to see still some some people around This talk is about a lot and plot tool, which I created
00:23
Yeah in the past month So the the contents of this presentation is a little bit who am I? What is this and then I hope to give some demos about? What I did So who am I? Yeah, I'm currently working at them collision as a software engineer
00:47
I've used programming languages Python C and rust and I've also worked on other projects like a PPI and risk Python and This presentation I will present about a lot and plot
01:03
As I think it's a good So what is this talk? This will be a little bit of a weird stock It's a bit fake But what I wanted to make was a tool which which you can visualize data from from systems
01:24
And this in a with live data and also zoomable So let me first start with a little bit of motivation. Why did I do this as mentioned this morning? almost all talks here the motivation is because Because you can and for me that's also the case it's enough but there's more motivation here
01:46
So there's when you debug systems. There's basically Yeah, two types of debugging you have either you you use a GDB and you start your program and you
02:02
Stop and step your program inspect variables Make variable watches and it's all very great and the tooling is also very good, but this does not always work If you have a controller for for example a motion stage you don't want to stop the The program in the middle because the stage will leave the leave the room which is not good
02:25
So we cannot break point the program So what do we do well, for example, you could do print F or you can toggle an LED Or you can monitor variable changes and I think we we can do better in this area
02:45
So what what is the system tracing involved? It's it's really inspecting the System in a test we built a whole machine and to do a test with it and typical With these kind of set ups is that
03:00
Testing is expensive time is limited. You have a lab. You're not the only one who has the lab A test takes time. So maybe it takes one hour. So in each day you only have eight attempts Well, that's not much Also resource are limited electricity water whatever you may use it's not infinite so you cannot
03:27
Repeat your test indefinitely Another problem with system testing is that tests are often not? As repeatable as you want them to be where as soon as you test you run the test once you run it again exactly the same
03:41
With the motion stage you run it once you run it again different result every time so one solution to this is Simply loss everything and analyze later you do the test once and you can you you have all the data that you that you need Some of my own experiences here with the
04:03
tracing of data I've I built in the last years several tracing utilities in Matlab in C sharp manuscripts in Python with different libraries and
04:22
I was not Yeah, really satisfied with this result Most of these libraries are intended for creating smooth graphs for publications, which is then not handy to use in a refreshing way or in a in a life data way
04:42
Well, maybe there's already an existing project. There are many projects. I have here a few but there are many more and After this talk, I hope to get Feedback from you The project I missed I want to make a list in the readme which lists many alternatives to to plot and
05:02
Visualize data from a life system. So KST plots from KDE. It's very good. It's very mature you can Load very big CSV files, but still it works on the CSV file. So you first have to create the CSV file There's another project plot juggler
05:20
Look looks promising. I have to try it out more Say rock is really cool user interface, but it's really focused I think on Getting if getting data from logic analyzers, there's even more tools GTK wave
05:43
But it's more for Digital logic so in that area then there's of course Ross operating system, which are 2d Also plotting tool, but then you're tied to Ross There's also a tool STM studio
06:01
Which allows you to monitor an STM processor And then there's of course whole slew of Web based tools, but I want to run it on desktop So that's a little bit of motivation. There's a lot but I Want something else? So, what do I actually want I
06:22
Want first of all, I want a desktop application. I want to do this offline in the lab And I don't want to use some clouds thing Of course having your data in the cloud is good because then you can share your data But for now, I want to keep it on my laptop. I Want to be able to zoom and pan and append data also log it on a file
06:42
preferably not CSV, but Yeah, probably the first prototype will be CSV and I want to get data from many different sources So usually you have a system with many different Processes in it. Well now go into the Program that I created for to do this a little bit about the design and how it's made
07:07
So top level is really simple ID We have a plot tool and we have a TCP connection and we have various systems we have a PC and we have a microcontroller and we send the data from the microcontroller and
07:21
the PLC system to the to the lot to the plot tool Okay So example use case yeah something like this We have a microcontroller and we have a Python script interfacing with this microcontroller and the Python script will center data over TCP
07:41
And next to that running in parallel we have an oscilloscope and which is also interfaced with for example the fisheye bush or Yeah, some Python library which allows you to communicate with the oscilloscope and this will also send the data to the plot tool One note is that the data is the data arrow is one-directional
08:05
And this allows for some interesting stuff for example We can we can Make despite the script interface it with a microcontroller and stream the same data to a disk and then later on We can load again this file from disk and again feed it via a TCP connection into the plotting tool
08:27
so then you have a sort of a Recording mechanism of what happens if you do this with all your data, you can replay it. So now for the first demo
08:57
So this demo I Created a GUI using rust and GTK which exactly this concept
09:27
this is a demo of having a Record which is a tool which loads with opens the microphone of this laptop and streams its to Standard output and then I pipe it into a script which will take this binary format and send it via TCP into the plotting tool
09:46
When I start this yeah, I have some prints of the data and In this this is the The rest tool and I can drag this data here and you see the data
10:01
Flowing in I can also zoom to the last 30 seconds or last 10 seconds or even Last seconds, which is nice If you do this Can make some noise maybe Some data, okay, which is good. I can also then you can pen this data and you can zoom this data
10:26
By scrolling into it. So this is handy. So this is good and we can also check the CPU usage of this program It's here. So we see that even if it's while it's ingesting a
10:46
Fair amount of data the usage is not that that large that's but You see the memory increasing but yeah, that's what you expect. It's audio data. It's not compressed with whatever
11:02
I will stop this for now. I will not keep it running but So what is the design of this log tool Primary idea is to have a reusable library
11:23
Which can be included in other applications for to have this plotting functionality and a showcase tool So what you just saw was the showcase tool it showcases. This is a nice graph and it works Yeah This is what it can do. So the graph of the program you can include in your own application. That's the design
11:45
so as you can see here, there is the Log and plot tool which will use the library and the library will have several things widgets which display the graph a charting Function or charting functions, which draw the graph
12:01
then a TSDB which stands for time series database, which will Have to the data in such a form that you can easily zoom and pan it and also a TCP server Which can receive the data notes that the arrows indicate the?
12:20
Dependency so you can pick for example the TCP server in the time series database and use only that part of the library So as several parts of this log and plot library, I will now mainly zoom into the time series database
12:42
How that is how that works? A side note time series database is very interesting. It's a whole new scheme of databases Specific for time series. So this is what I made was not really a time series database as such but ideas are similar
13:04
some Properties of the time series database. I made some assumptions data can only be appended and
13:20
So I will now go into a little bit of how this signal data is structured in memory so what I did was create a sort of a b3 and And group the samples in batches and keep metadata for each batch. So what does this mean? It means we get the data in as separate samples. We split it into batches of a certain amount of samples
13:45
For each batch we have a Set of metadata such as when was the first sample received one was the last sample received. What was the minimum value? What was the maximum value and what was the mean? What was the standard deviation these kind of things we keep track of that for each batch
14:04
Next we organize these batches into a b3 Which is a tree structure with a root node intermediate node and leaf nodes. So the samples will be in the leaf nodes and each leaf node will have
14:23
Metadata about this sample about the samples, but the intermediate nodes will also have this metadata So now what can we do when we visualize this data? So when we are really zoomed out We will only
14:42
Go into the tree to a certain level we will not go all the way to the Raw samples, but if you're really zoomed out, we will only go to halfway the tree so when we draw the
15:00
The plots we draw the access and we draw the curves depending on our zoom levels and we draw legends So what does this mean in zoomed out form when you zoom out into the plots you will see the min max mean and standard deviations Like is displayed here. So the you see the mean line in the middle and then the
15:22
transparent Region indicates the min max of this signal and if you remember the tree structure The previous graph was made From intermediate level nodes, so not really zoomed into the leaf nodes
15:40
When we zoom in you see individual data points and when you see something like this Then we're really at leaf nodes, but only at the specific part you zoomed in of the data So this is in a nutshell how this zooming functionality works so As I already said I demoed the first demo with the Rust implementation
16:04
So there's actually two implementations one in Python and one in Rust. So why did I did it like that? Python is handy. If you want to try out a new ID Python is ideal because it works really fast But the performance is maybe not that good. So that's why I also made a Rust version which compiles really to machine code
16:25
Unfortunately, the Rust version is made on GTK and the Python on PyQt, but it's good to have two redundant implementations to Try out things In the Rust version I used Tokyo RS Crate that is a
16:42
TCP or a Networking crate to handle TCP connections and also made a C wrapper around the Rust library So you can use this from C code you can link to this library to for example
17:00
Send a data from your program. So I will now demo other stuff See so we already saw the audio signal, which is nice but What I also
17:21
quickly added to this program was a sort of Was the the rendering time of the graph itself as a signal because we can render signals So in this graph, you will see the the rendering time of this plot. So the the rendering time is always around
17:41
four milliseconds so yeah, you can also zoom in on this an other demo that I Have is This is a batch download demo. So say you have made a recording and you want to just look into this data
18:05
See what happened You can make a Python script we will which will load in the SQLite database CSV file txt file whatever you have and send it to this tool and this script will send 10 million samples to the plotting tool
18:23
So it will it will take some time But not not that long So we can delete it and again we can zoom this data. So these are 10 million data points and again, you can zoom it as before and this is another name of it generates some some noise and
18:51
It will send to two traces drag tutor to traces you can also
19:01
Drag multiple traces in the in the program So this is the rest version I can also demo the The Python version So this is the the the Python with pikute version if you prefer to use Python you can use this one
19:31
you can switch here to You can switch to multiple views. I made this view that you can have multiple
19:42
plots But I will now use one plot and you can also view it here And I can do that I can start the same demo as before and send the data not to the rust Program, but to the Python program and usually you should also benefit of having the scripts which interact with your hardware separate from the
20:07
GUI to visualize it because now I have the same signals But in this in this Python version and as you can see I implemented in the Python version already something Extras, which is that this color indicates that these signals are sort of life. So the the data was arrived
20:24
And recent And you can also drag them here and zoom in So you can see that the Python version version actually also performs pretty reasonable pretty good But the memory consumption is rather rather large
20:57
Yes, I wanted to give an arm demo, but I'm not sure
21:01
If there's enough time for it But let me explain the ID so I made a I made an another example which uses this board It's microcontroller board with an STM 32 on it just to really get hardware and lock this data into the to the plotting tool
21:24
The setup is like this We have a microcontroller and an ST link which is on this board and then I made a rust client which interacts with this ST link and send this data also to this plotting utility and I wanted to highlight this this is made by made available by Armin is
21:46
present in many Or arm course. It's the traceport units the data watch point and trace and an instrumentation trace Blocks which allow you to do this tracing kind of thing
22:00
So not to stop and continue debugging but you can configure these devices to Monitor memory regions and whenever this memory region changes it will generate an event with the program counter so the The part of your program which changed the memory and the memory value and a timestamp So this is ideal for tracing and not holding the system. So the tracing strategy here is to
22:29
Use the staling and Connect to this as tailing using the art our USB library, it's a rust crate to interface with USB and
22:40
Send this data again to the tool This is the sample program I made really small loop with just one volatile variable so that I have a memory access which is changed So you have to believe me that is in this board. I will now check if this works. I have to
23:11
Make sure that I have to start the plotting tool first because that will open a TCP server. Yes
23:38
So we see here the the signal which is coming from this board
23:42
so it will send the data and this board is now running in free running modes and we capture each time this variable a is written and with the data watchpoint and trace unit and I Gather this event and send it to the to the plotting tool. That's all
24:12
Thank you for your attention. The link is here. So all kind of feedback is welcome. If you want to add something Please do so
24:22
Thank you