Scientific Visualization with GR
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 46 | |
| Anzahl der Teile | 119 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/19986 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produktionsort | Berlin |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
EuroPython 201446 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
PASS <Programm>Visuelles SystemGruppenoperationPhysikalisches SystemVisualisierungSchreib-Lese-KopfFormale SprachePhysikalismusBildverstehenComputeranimation
00:32
SoftwareVisuelles SystemProjektive EbeneFramework <Informatik>DifferenteMereologiePhysikalisches SystemVorlesung/Konferenz
01:08
VideokonferenzPlot <Graphische Darstellung>Metropolitan area networkStandardabweichungOverhead <Kommunikationstechnik>MAPHardwareGrenzschichtablösungLokales MinimumSchnittmengeLucas-ZahlenreiheCodeÜbersetzer <Informatik>GraphikprozessorFront-End <Software>Treiber <Programm>Klon <Mathematik>MailboxDistributionenraumReelle ZahlGroße VereinheitlichungGammafunktionUniformer RaumAlgorithmusComputerspielDifferentialgleichungInformationSimulationSoftwareCodierungWürfelInformationsmanagementPlotterFrequenzProgrammbibliothekMAPVideokonferenzEchtzeitsystemPhasenumwandlungTaskÄquivalenzklasseAggregatzustandAlgorithmische ProgrammierspracheAnalytische FortsetzungBildschirmmaskeGeradeGrundraumKardinalzahlLokales MinimumMereologieNumerische IntegrationPhysikalisches SystemPortabilitätRechenschieberRechenwerkVisualisierungZahlenbereichGüte der AnpassungServerInternetworkingParametersystemCASE <Informatik>Zusammenhängender GraphPunktspektrumCoxeter-GruppeVollständigkeitPunktDichte <Physik>Dimension 3SchnittmengeOffene MengeLesen <Datenverarbeitung>Kartesische KoordinatenInteraktives FernsehenEinfügungsdämpfungDatenstromData MiningSkriptspracheFramework <Informatik>SoundverarbeitungElektronische PublikationSchlüsselverwaltungDifferenteRechenbuchSystemplattformLoginKurvenanpassungMultiplikationsoperatorRechter WinkelDienst <Informatik>Interface <Schaltung>Demoszene <Programmierung>Figurierte ZahlMehrkernprozessorEinsCodeHardwareStreuungAusdruck <Logik>ComputersimulationFunktionalLeistung <Physik>Treiber <Programm>Graphische BenutzeroberflächeDistributionenraumMultifunktionFlächentheorieQuellcodeMixed RealityStandardabweichungKeller <Informatik>Computeranimation
09:34
MultiplikationBimodulParallele SchnittstelleProgrammschleifeGraphikprozessorSimulationMetropolitan area networkRechnernetzCodeComputerspielPhysikerSimulationStreuungEchtzeitsystemRegelkreisGrenzschichtablösungGlobale OptimierungMereologieSpeicherabzugWinkelZahlenbereichDatensichtgerätCASE <Informatik>AdditionBildschirmsymbolKernel <Informatik>Kartesische KoordinatenArithmetischer AusdruckBefehl <Informatik>FlächentheorieWeb SitePartikelsystemMultiplikationsoperatorZweiHardwareExperimentalphysikDimensionsanalyseKomplex <Algebra>Physikalische TheorieSchnittmengeFramework <Informatik>Suite <Programmpaket>
13:47
Metropolitan area networkPhysikalischer EffektCodeGammafunktionPortscannerReelle ZahlFunktion <Mathematik>VideokonferenzSchaltnetzSimulationSoftwareStreuungProgrammbibliothekTypentheorieEchtzeitsystemIntegralProgrammierumgebungFunktionalGeradeVirtuelle MaschineVisualisierungParametersystemCASE <Informatik>DistributionenraumPunktKartesische KoordinatenBeobachtungsstudieSkriptspracheBefehl <Informatik>Framework <Informatik>MinimumVerkehrsinformationQuellcodeAuflösung <Mathematik>Demoszene <Programmierung>KoroutineCodeRelativitätstheorieWärmeleitfähigkeitAggregatzustandMaßerweiterungRechenschieberResultanteZahlenbereichEnergiedichteDateiformatSchlussregelOntologie <Wissensverarbeitung>Mechanismus-Design-TheorieComputeranimation
16:34
Auflösung <Mathematik>Leistung <Physik>Front-End <Software>Lie-GruppeDesintegration <Mathematik>Web SiteDokumentenserverSoftwareMailing-ListeWort <Informatik>TermVersionsverwaltungKontrollstrukturBenutzerfreundlichkeitRelativitätstheorieSoftwareIntegralAutomatische IndexierungElementargeometrieBildschirmmaskeBinärcodeLeistung <Physik>Physikalisches SystemRechenschieberTermVerschlingungExogene VariableAutomatische HandlungsplanungParametersystemCoxeter-GruppeDistributionenraumPunktWort <Informatik>Lesen <Datenverarbeitung>Kartesische KoordinatenMailing-ListeSkriptspracheFramework <Informatik>Abgeschlossene MengeEndliche ModelltheorieBenutzerbeteiligungKoroutineVisualisierungVersionsverwaltungRepository <Informatik>KontrollstrukturBimodulWeb SiteFront-End <Software>SoftwareentwicklerComputeranimation
19:23
CodeFolge <Mathematik>SoftwareWellenpaketVektorraumIntegralBefehlsprozessorInterpretiererVisualisierungZahlenbereichVersionsverwaltungNichtlinearer OperatorNeuronales NetzProzess <Informatik>FehlermeldungATMNotebook-ComputerFramework <Informatik>SchnelltasteDiskrepanzDifferenteNeuroinformatikAuflösung <Mathematik>Demo <Programm>AnalysisComputerspielEntscheidungstheorieHybridrechnerInverser LimesLeistung <Physik>Virtuelle MaschineBasis <Mathematik>Wort <Informatik>AblaufverfolgungRechter WinkelMobiles InternetVorlesung/Konferenz
23:12
Uniformer RaumGoogol
Transkript: Englisch(automatisch erzeugt)
00:15
Joseph Heinen is the head of the scientific IT systems group at the Forschenzentrum Jülich in Germany
00:22
and he will be giving a talk on the visualization system, on the GIS visualization system. Okay, thanks. You're welcome.
00:40
Okay, first of all, thank you for coming here this morning. My name is Joseph Heinen and together with my colleagues, I'm working on different projects at Forschenzentrum Jülich. Most of them are visualization systems and I'm proud to have the opportunity to give this talk here at EuroPython about GR, a framework for visualization systems.
01:08
So let me start with a question. Who is already using some scientific software with Python such like Matplotlib, Mayavi, VTK or... Okay, that's a big number.
01:20
As mentioned, I'm working at a research company and in the past years, it turned out that there is a growing need for better and faster visualization software. Especially, scientists need easy-to-use methods for visualizing two and three-dimensional datasets, possibly with a dynamic component.
01:46
And they want to create publication quality graphics and videos for the publications probably in the internet. And they want to make glossy figures for high-impact journals or press releases.
02:02
At first glance, those methods don't seem to be very challenging but we are talking about it later. There are a lot of scientific plotting methods we need such as line bar graphs, curve plots, scatter plots, all these things you see on the slide here.
02:23
In principle, this is nothing challenging and there are dozens of solutions for all these kinds of plotting methods. There are also powerful software libraries for scientific applications in Python. Those listed here on this slide are the most popular ones, I think. Maybe I've forgotten one.
02:45
But we all know Matplotlib which is the work host and the de facto standard concerning graphics in Python. And there's even Myavi for three-dimensional applications, Myavi which is very powerful and based on VTK.
03:02
And it offers an application interface called MLAB which can be used in your own scripts. There's VTK, it's very versatile but it's difficult to learn because it's a very low-level system. And we have tools like Wispy and OpenGL which are both very fast and which
03:25
are limited to 3D and which are really the lowest-level APIs for graphics with Python. And there are also some GUI tools just like QWT with its corresponding 3D equivalents.
03:41
And the problem with this is that they are currently unmaintained, at least for what my information is. So there are some problems so far and the main problem I think is that the 2D world and the accelerated 3D world are separated. You won't find a tool which provides services for both 2D and 3D graphics.
04:09
And another problem is that some graphics backends only produce kind of figures. So it's not possible to present continuous data streams from live sources.
04:22
And also I've made the experience that there's only a bare minimum level of interoperability. So user interaction is somehow limited with these tools. Also if we are talking about analyzing large datasets, we often see that there's only a poor performance.
04:47
And also these APIs are partly device and platform independent. So your own scripts will suffer from some system dependencies after the time.
05:03
So let's Python get up and running and push for Python. There is a very nice distribution which has been introduced to you in the keynote this morning. It's called Anaconda and I would really like to recommend this distribution as it's very easy to install a complete scientific Python stack.
05:31
But I think we need something more. For example we need some more performance and this can also be achieved by Anaconda add-ons.
05:42
For example NumPy, which is also mentioned this morning, which is capable of accelerating Python applications based which contain NumPy code even on GPU hardware or multi-core processors.
06:01
And I will give some example later. There's something more. We also want to achieve more graphics performance and interoperability. And for this purpose I would like to introduce our GR framework, which is a universal framework for cross-platform visualization.
06:22
And the main key points are that it has a procedural graphics backend so you can really present continuous data streams. And it has built-in device drivers so you can visualize both 2D and 3D scenes in one canvas. And there's a very good interoperability with GUI toolkits so you can establish a very good user interaction.
06:48
And as you can see in the bottom part of the slide it's also very easy to install. So this would be our complete scientific Python distribution.
07:04
I think we have everything we need. Especially we have more performance and interoperability. So let me give some examples how this looks live. You can see here a numeric simulation of a damped pendulum.
07:24
The calculation is done in the ARCA4 function. You can see which is simply a numerical integration of this differential equation. And you can see that you can mix graphics with text formulas.
07:41
And you can do all these things live while your script is running. You don't have to produce figures or something like this. The same works for 3D. You can see it here. In this case, visualization is done with an API which has been written by a colleague of mine, Florian Riem.
08:03
And he has written an OpenGL layer for GR, which is called GR3. And you see it's very performant and it does its job. You can even visualize live signals from Rave files or from the microphone.
08:28
And with lock X-axis. And this all runs in real time. These are all things which are very hard to realize with other tools. You can do this also in 3D.
08:45
I just pushed the audio away so you can focus on the graphics. So the frequency spectrum is, in this case, visualized by a surface plot which is realized with OpenGL and it's that fast as you could see.
09:05
You can also produce graphics with user interaction. You can see here an MRI application which renders some MRI data through a marching cubes algorithm which is part of our software.
09:26
And which can be rendered very very fast and moved with a mouse. So let's talk again about performance. We not only have some need for more graphics performance but also for numerical performance.
09:48
And as mentioned before, there's something called Number which is part of Anaconda but you also can install it for your own. And there's Number Pro which has some additional features.
10:02
It's part of Anaconda Accelerate which cost a few bucks, I don't know the actual price. And it's capable of calculating NumPy expressions on the GPU so you can write your own GPU kernels in Python.
10:23
And it's a very nice tool and it's worth to look at this software. And there are even other tools like CooBlast, CooFFT, CooRant but those tools are just dedicated to CUDA hardware.
10:41
So in this case you can see how you can profit from such software. You see particle simulation which is very slow, currently running at 3 frames per second. And just by adding some decorators and an import statement for sure, you can increase the performance by times 15 I think.
11:06
So you don't have to change your code and you can speed up your application enormously. This is calculated in real time, this would not be possible with poor Python.
11:21
If you run this simulation in Python, I think each frame is about 3 seconds. In this case it was parallelized or vectorized. I have several examples in our demo suite. Just take a look at the website and you will see how the different optimizations work.
11:44
So let me introduce some of our success stories. We have integrated our software in several of our applications. We are working both for experimental physicists and for theoretical physicists.
12:02
And this is something for our instruments. It's a live display for a small angle neutron diffractometer. And as you could see, you can set the region of interest and the surface is generated in real time.
12:23
You can rotate it, you can flip the axis and there is even more. And all this can be done in real time. So this is another example. Here we are processing a huge data set and it's also done in real time.
12:43
And this was formally done by a proprietary solution. And with a GR framework we could embed this into a QT4 application. Which was a replacement for the existing solution. And which is much faster, which can produce movies and all these funny things.
13:05
There is another example here. NICOS. It's a very complex network-based control system which is used at Froshungsdag dimension. In Munich for all the instruments which do neutron scattering.
13:25
And in this case we replaced by QVT, QWT application with a QTGR application. And it was much faster, it was more responsive and it had some additional features which we didn't have before.
13:48
So this is a case study to see how fast you can simulate data. Born is a software for simulating neutron and x-ray scattering. To compare it, it was a replacement for Matplotlib at this point.
14:10
And it uses a single call, it's the line just at the bottom of the left side. And if you look at the old code, if you compare the old source code with the new one.
14:22
Well that's only one line and an export statement to generate a movie for example. So it's not such complicated to produce movies with a GR framework. So what are the conclusions?
14:41
The use of Python with our GR framework and Numba and perhaps Numba Pro extensions. Allows the realization of high performance visualization applications both in scientific and technical environments. And the GR framework can seamlessly be integrated into any Python environment.
15:06
I would suggest to use Anaconda. The integration is simply done by a C types mechanism. So you can also use it in your own Python distribution.
15:20
And the combination Conda and Anaconda provide a very easy to manage and ready to use Python distribution that can be enhanced by the use of our GR framework. Especially with its functions for real-time or 3D visualization functions. So what's next?
15:44
We are not far from implementing molecular dynamics package. And which will produce such results. We have already all this stuff written in C and simply have to write some simple reports which will then be integrated into our GR framework.
16:04
And with this framework you will be able to do things like this here. This is a simulation which has been calculated on a very big machine and the data is read with a simple Python script and then rendered with a GR3 library.
16:21
You can then export this scene to for example Povray and produce a high quality graphics like shown on the right side of the slide. And you can even do this in highest resolution if you give the correct parameters to those routines.
16:41
And you can see here it's a realistic presentation of a DNA. So what are our future plans? Well we have thought to combine the power of Matplotlib and GR. And we think it should be possible and the basic idea is to use GR as a Matplotlib backend.
17:07
So this would speed up Matplotlib and all your Matplotlib scripts would profit from this speed up. I think it's possible. We didn't start this development but I think there's a good chance that we get these things running.
17:32
And there are even more challenges. You learned about Bouquet this morning and I think this should also be possible.
17:42
Once we have the Matplotlib integration it should also be possible to connect those scripts to the Bouquet backend which Travis mentioned this morning in the keynote. At this point I think we should talk to Travis to cooperate.
18:02
On this slide you find some resources. There's a website for our framework. There's a Git repo. It's hosted by package index. We even have first Beanstar binary distributions for the GR framework.
18:22
And the talk should be online on this link which you can find out later. So some closing words. Maybe you hate me after this but I think that's important. I think that visualization software could be even better if the prerequisites for an application would be
18:45
described in terms of usability, responsiveness and interoperability instead of a list of software with module dependencies. We should use native APIs on the different systems instead of GUI toolkits and release updates should not break version compatibility.
19:06
This is something that I have observed very often. So let me end up and thank you for your attention.
19:24
Thanks Rami for this great talk. Are there any questions? One of the features of Matplotlib that I find very convenient is its integration with IPython notebook because I can play
19:43
with the visualization before I integrate it into some application or save a high resolution copy for publication or something like that. So I wonder if is GR framework compatible with IPython notebook and I use it from there? Right now it's not but because there is some discrepancy on the one side we are
20:04
talking about immediate mode graphics and with IPython notebook there is just a sequence of commands. Maybe if we get our Matplotlib back and running as we consider that it could work
20:23
then it might be possible to use it in IPython but I am not sure about this. Thank you. We will do our best. Thanks. I am doing a lot of training of neural networks inside CYthon almost completely outside of global interpreter lock. Would it be possible
20:48
for me to bind to CIPIs or CYthon IPIs that are excluding global interpreter lock or do I have to bind back through Python then? As I want to visualize the training of the network during the training process I think this would be really cool for that.
21:04
I am not sure about this. I think we have to talk about this after the session. Sorry. Are there any more questions? Okay. I have a question. When you use the Specterize decorator and you use the same code in the limited or the basic number
21:28
version will this just do nothing or will it complain with a name error or something or import error? Is the Specterize decorator available even if it does nothing in the basic number version?
21:45
So you mean if it is not present on your machine? I mean for example you have the Specterize which I think I understood only works for the pro version and if you have only the basic version. The difference is that the pro version is capable of pushing those LLVM code on
22:07
your GPU and the public version is only capable of parallelizing on your own CPU. So pro version is only needed if you want to use your GPU for the computing of NumPy operations.
22:24
What I meant is if I get code from you which has the Specterize decorator and I only have the basic version installed will it just not vectorize but otherwise ignore the vectorize decorator? No this would not work. So if you want to try those demos you really have to purchase the pro version.
22:48
But there are a lot of other demos which don't depend on the pro version. Thanks a lot. Meanwhile no questions have been added.
23:04
So thanks again.