Interactive (EO) data visualization in the web
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43498 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Software testingLevel (video gaming)
00:24
Service (economics)Interactive televisionVisualization (computer graphics)Computer fontRaster graphicsPlot (narrative)Web browserVolumenvisualisierungMusical ensembleShader <Informatik>Artistic renderingOpen setStandard deviationClient (computing)Modul <Datentyp>Component-based software engineeringScripting languageJava appletLibrary (computing)Scale (map)SoftwareCASE <Informatik>Component-based software engineeringMultiplication signSoftware developerBitLevel (video gaming)Artistic renderingWeb browserLibrary (computing)Fitness functionEndliche ModelltheoriePoint (geometry)Projective planeOpen sourcePlotterRaw image formatVector graphicsClient (computing)Shader <Informatik>Raster graphicsModule (mathematics)Software repository
04:08
Java appletScripting languageLibrary (computing)Raster graphicsScale (map)SoftwareVisualization (computer graphics)Web browserRange (statistics)Public domainMatrix (mathematics)MathematicsAlgebraLocal ringRegulärer Ausdruck <Textverarbeitung>SharewareComputer configurationComputer chessArtistic renderingWeb browserSoftwareSharewareSign (mathematics)Functional (mathematics)Public domainPixelGraph coloringWell-formed formulaOperator (mathematics)Musical ensembleSoftware developerCombinational logicMereologyChief information officerLibrary (computing)Internet service providerComputerWeb 2.0Formal languageScaling (geometry)Computer fileServer (computing)MathematicsVolumenvisualisierungMedical imagingCASE <Informatik>TouchscreenModule (mathematics)Raster graphicsAlgebraic functionParsingPhase transitionAdditionShader <Informatik>Lecture/ConferenceComputer animation
07:28
SharewareMathematicsVolumenvisualisierungPlot (narrative)PlotterRAIDMedical imagingElement (mathematics)Graph coloringScaling (geometry)MereologyGoodness of fitMathematicsoutputDefault (computer science)Flow separationPlotterClique-widthSharewarePublic domainOpen setDigitizingWeight
08:55
SharewarePlotterFunctional (mathematics)Level (video gaming)Open setSharewareEndliche Modelltheorie
09:28
SharewareExpressionGraph coloringMusical ensembleScaling (geometry)PixelMedical imagingSharewareMathematics
09:58
Polygon meshSharewareMedical imagingDifferent (Kate Ryan album)Well-formed formulaPublic domainGraph coloringGoodness of fitSubject indexingMatter waveMusical ensembleScaling (geometry)Computer fileWordView (database)Library (computing)Mereology
11:09
Plot (narrative)SVGRaster graphicsTransformation (genetics)Object (grammar)Local GroupLimit (category theory)Physical systemArtistic renderingOpen sourceElement (mathematics)Uniqueness quantificationReading (process)IterationPixelShader <Informatik>RectangleShape (magazine)StapeldateiArray data structureAttribute grammarGlass floatPoint (geometry)Price indexRadiusFluidClient (computing)Computer-generated imageryTerm (mathematics)Software development kitLibrary (computing)Point (geometry)Right angleType theoryExpressionLatent heatAdventure gameVolumenvisualisierungTouchscreenPrimitive (album)CalculationParameter (computer programming)ScatteringCoefficient of determinationCircleShape (magazine)Poisson-KlammerProjective planeBitDistanceFamilyComputer wormFirst-person shooterInstance (computer science)Scaling (geometry)Dot productInformationPlotterSymbol tableMedical imagingAttribute grammarMultiplication signRectanglePosition operatorWeb 2.0Object (grammar)Link (knot theory)Scripting languageLine (geometry)Computer graphics (computer science)Interactive televisionForm (programming)TriangleVector graphicsGraph coloringWeb browserCodePixelCombinational logicInheritance (object-oriented programming)Term (mathematics)Physical systemTrailGroup actionCASE <Informatik>Artistic renderingScalabilityKeyboard shortcutShader <Informatik>Characteristic polynomialRaster graphicsDifferent (Kate Ryan album)Graphics processing unitDescriptive statisticsSquare numberAddress spaceLimit (category theory)Pairwise comparison
20:27
Service (economics)MassProjective planeArithmetic meanUMLLecture/Conference
20:48
Rule of inferenceParameter (computer programming)Gamma functionInteractive televisionWebsiteParameter (computer programming)PlotterGraph coloringScaling (geometry)Projective planeInformationShader <Informatik>Object (grammar)Array data structureProduct (business)Lecture/ConferenceComputer animation
22:03
Different (Kate Ryan album)Server (computing)Product (business)Moment (mathematics)CuboidRight angleRevision controlWindowOpen setTrailPlotterElement (mathematics)Artistic renderingPresentation of a groupLevel (video gaming)Software bugMeeting/Interview
25:08
Plane (geometry)
Transcript: English(auto-generated)
00:07
I suggest we continue with our second talk of this session by Lubomir Butsek and Daniel Santillan about WebGL magic and scientific data. The stage is yours.
00:27
All right. Hi. Thanks, Joachim. So I think, yeah, we've heard a little from Ivan, maybe a bit more enthusiastic than me. Let's see. I'll try. That we can actually use the browser
00:41
to, yeah, to do some nice things. And we're going to talk about two of these things. So, yeah, we've been working and looking into some of the things that he has also been actually presenting, so actually showing scientific data directly in the browser.
01:03
So in this respect, so we will be talking about, so we called it Plotty and Graffley. I think having a Y at the end of the library makes it much better. And so one takes a bit the approach of looking at raster manipulation
01:25
and this rendering in the web browser. And the other we worked with vector graphics and how we can actually move away from that and do things in the shader
01:41
to actually get some performance. So maybe from us, just to say, so we're from EOX. Maybe you've seen some talks about raster manipulation from Joachim Marchetti. Also Fabian tomorrow and our colleague Peter will be talking about some other libraries.
02:06
So we work mainly, basically most of our developments are free and open source. I think our main client being ESA, I think it's really nice to see that we can move like that and do our developments like that. And what we have seen is that as we go along the years
02:27
and go from one project to the other, we're actually trying to, we see that modular development, so instead of doing big things that usually don't fit exactly the use case of every person,
02:42
we're trying to go more piecewise, so just have small modules that are kind of interchangeable and can be used by everybody. I think we're seeing this more and more, and we're seeing that our GitHub repos are being used, I think,
03:02
and we're seeing some pull requests because they are nice individual components that can be really used independently. So we see this like Geotiff.js and Plot.js and roughly maybe at some point, which I'll present a little bit more.
03:23
So yeah, let's say that's a bit about us. I'll let Lobo talk a little bit about Plot.js and maybe, I think this was, the initial stages were presented already, the last Phosphor-G.
03:44
So yeah, we'll talk a little bit, he'll present a little bit about what new things have come along and what he has also helped implement or integrate at least. Okay, hi everybody. I'll first shortly introduce Plotty.
04:07
Plotty is a really lightweight library for putting one-band raster data on a canvas. That's as simple as it can get. As Daniel said, it has already been introduced three years ago.
04:27
Since then, the development has gone to a phase where everything that we want Plotty to do, it already does. And the main addition has been moving to the ES6 module style to be able to import more easily
04:46
and to go with the standard movement and also adding the raster algebra functions to be run directly on the shader. As being said, the purpose of the library
05:01
is to quickly set a color scale to the data that we give it. It tries to use the WebGL if possible, which all new browsers definitely can. It also provides a fallback to standard software rendering if the WebGL is not provided.
05:21
However, in most of the browsers this will not happen. The motivation to create the library some years ago was to try to colorize the raster images in the browser because in some cases it might be much faster, in some cases not.
05:44
It always depends on the use case and on the speed with which you can actually get the image to the user's browser. Previously all the renderings were done by us in WMS with some preset styles,
06:01
but then every change of the style needed a new fetch on the server. Another option emerged as getting the full file by WCS and then try to render it by WebGL. Another option, which emerged a few years earlier, is with GeoTIFF.js to get the part of the GeoTIFF parse
06:30
and then only the relevant data or relevant bands will be visualized. So what the library can do is basically only these things.
06:44
It should be enough for the use case that we provide it to. So the new thing is the dataset combination where for each pixel given the math formula using the plus minus signs and the normal mathematic operations
07:05
and also some functions like sinus cosinus, basically anything that underlying GLSL shading language can provide, compute a value and this value will then be colorized based on a color scale.
07:23
I've actually prepared a demo which then I figured out that is not working, so sticking back to the screenshots, good old screenshots. First is a static demo which on the click just showed some arbitrary data.
07:41
This is a sinus function, so the first part actually only produces our values and then the plotty plot chart is created. It targets the canvas element, it sets the data and it sets the width and the height. That's all the input that it needs.
08:01
You can set the color scale or use the default one and then you render through the canvas. If we want to show it on some more sensible data, another image is showing data from the synthetic aperture radar where several input elements
08:29
can trigger the change of the color scale like here where the change is instant because we are rendering everything in the WebGL.
08:42
We can set the min and max, basically select the domain which can then be either clamped or not, which also has an influence on the resulting image of course. If we don't want to directly render only image to a canvas,
09:00
we can use the rendering to a canvas for example in open layers which allows to render over the open layers canvas where the demo that you see is rendering digital elevation model on open layers base map and valleys and the mountains are cut using the clamp functionality.
09:31
The last demo is showing Sentinel-2 RGB near infrared image where here all the first three bands RGB are used
09:41
where just the intensity of the image is computed for each pixel which means R plus G plus B divided by three and a simple color scale is applied on the resulting values. By simply clicking a button you can change the math expression that you are applying on the image
10:04
which for example here is good old NDVI index and setting a domain and for example different color scale. A quite similar image can be obtained using different formula where one of many formulas for detecting chlorophyll A is used
10:25
where again the three bands, here the RGB bands are used and there are specific wavelengths and again a domain is set on this resulting image. Those were just a few really simple examples what the library can do
10:47
and there are several more for example you could use cesium to visualize a 3D view and then plotty can be also added on top of that to visualize some nice behavior
11:04
or some motion. With this I am actually giving word to Daniel to talk about the second part of our attempt. Thanks. So going just for a second back maybe
11:24
hopefully no need to go to the fragment data directly so you can use the library to use some expressions hopefully a little bit more easier than having to dig down deep into what you are doing.
11:43
Next library is roughly the idea was not necessarily EO data that is why it is in brackets in the title basically we have seen in many projects that we need to show a scatter plot
12:04
and we have seen a lot of screenshots of showing timelines and things like that so points connected with lines and when we began with this adventure we basically said ok it makes sense to use scalable vector graphics in the browser
12:22
easy you can manipulate them, interaction, you have everything that there is to have but we have seen that this is not always the solution we need so there is like a crazy amount of libraries that allows you to do this
12:41
that they are sometimes based on D3 or not, they are their own thing and they are usually like you can generate so many types of plots but when it comes to solving your specific use case if it is not matching exactly what the library does
13:02
it gets complicated quite quickly and this is what we were seeing and especially one of the things that was a main issue for us is that we were trying to render more than 6000 points at least which was really difficult to do with SVG
13:23
the browser resources for that is ridiculous so we began with the idea like oh ok shader can do everything right? so let's do it in the graphics card I mean basically it is not inventing anything new but I think we have brought multiple things together
13:43
that work really nicely together and I think it is quite a nice thing to have so this is something at least that we needed so maybe let's compare lecture graphics against rasta so what does the one thing give you and the other not?
14:03
so with SVG you have like really nice manipulation you have groups, you can transform complete groups you can do a lot of really nice animations and really a lot of nice stuff but as I said one of the main limitations is really
14:21
how much stuff you can use so the system gets quickly overwhelmed so you have all of this as nodes and this has to be kept track of so that is quite intense and the rasta on the other hand once you render it you are basically saying this pixel is this color
14:40
and that's it like there is no real way of interacting with this there are some ways of doing it but no trivial ways so it is difficult to interact and to manipulate but it is like super fast usually depending on how you use it but yeah that is the idea
15:03
so what did we do? we thought ok let's try to win back the things that we lose using rasta so basically how do we address and get information of what we are rendering
15:20
and what we do is this is often done in computer graphics to just render the image twice and once you render what you are showing to the user and the other time you use a unique color you can go insane
15:42
the human eye does not differentiate but if you are picking the color you have like 1 billion combinations 1, 2, 3, 16 million so you can render a lot of things and just use a different color for every point
16:00
depending on what you are rendering and then you have this raster and when you move the mouse over you just pick the color and use the color as a reference to your actual attributes so you keep the interactivity there here is just the code for reading the pixel value
16:22
which is super fast you can do it on mouse over so that's no problem the next thing we combined is it would be nice to just show rectangles as in the plot but as soon as you are showing 1 or 2 parameters
16:42
you want to differentiate them somehow so you need different forms you need like triangles and squares it would be quite painful or at least not as efficient to do this as primitives so actually you are rendering this
17:01
so again the fragment shader to the rescue you can do some pixel manipulation which works great so we use just the dot primitive we are rendering so there is some really nice
17:21
descriptions of what can be done and great resources linked here which gave a lot of inspiration and apart from that you can use instancing which again even further so you are using an instance of the same object
17:42
with different characteristics I don't know how deep we want to go into this so what you do is you basically define your information let's say in JavaScript generate some attributes of what you need like the position, the value, size and symbol
18:07
let's say this would be 5 attributes 5 attributes you have for each of the dots you are representing and then you bind this create a buffer and pass it say the location for the shader
18:22
and then basically pass it to the shader this is how you define and pass the information to the shader instancing is WebGL 2 but there is compatibility for WebGL 1 where you can try to get the extension
18:42
and if the browser supports it yeah, so I think I will not go more into detail if there are questions just feel free afterwards so what can you do? so how long time? just to one minute alright, so I'm
19:00
alright, so this is taking longer than expected so this is basically what you can do in the shader you can do some calculations of the distances and then you can create different shapes for each of the dots you are rendering so this gives you like for example this would be how you render a circle
19:22
using just the let's say dot primitive which would be a rectangle and this is basically how it looks like so here it's an example of 3 different parameters we are rendering with the scatterplot and they are all just dot primitives
19:40
but just in the fragment shader we apply some calculations to manipulate each of these points and another neat feature that we combined is let's say debounce rendering so instead of rendering for every manipulation that you do
20:00
you introduce a debounce rendering that you actually use a fixed image and manipulate the image like scale it and transform it and once the movement is done the rendering is done again so these 3 things are combined to create a really strong and flexible
20:23
let's say not only scatterplot but mainly scatterplot rendering library which we think it's quite nice to have and we have used in many projects already alright, so depending on the questions
20:40
maybe in the 5 minutes I can give like I'll just give like a super short to give you an example of what this means sorry
21:00
so let's say basically this is it you can interact with it you can zoom in some out and again you can for each of these values you can get information on every point
21:20
and as you have the data here you can play with filtering so there is a lot of stuff going on as we are doing it directly on the shader again you can play with the color scale which is, you can change the color scale which parameter is rendered so yeah, a lot of possibilities
21:44
yeah, that's basically it so feel free to give it a try if you are using some plots the data can be loaded just as an object and arrays so it should be fairly easy to integrate to some of your projects alright, so thanks
22:04
yeah, thanks for the great talk I know there has been a lot of work and sweat going into it so, are there any questions from the audience?
22:21
of course is this ready for production? yeah, you reserved this one from the previous, right? is this ready for production was the question? yeah, for sure depends on your production
22:40
no, I mean, it works will there, might there be back somewhere? for sure but, I mean, we use it in a what, in ESA we call pre-operational which is important because it's on our servers
23:00
and not on their servers and yeah, I mean, it has had a good let's say, use from different users and some bugs have been found and we try to keep track and solve the things that appear yes, you try to fix the 95%
23:21
and the 5% left yeah, you try to fix when it pops up okay, next question
23:41
no? don't be so shy yeah, so the question was if it integrates well with other tools like, for example, open layers
24:03
difficult to say I mean, in principle you give this thing canvas element and that's it, like, it doesn't need more the question is I don't know, like, how you combine a plot with a map
24:20
I guess this thing could render over the open layers canvas maybe there's some conflicts on which rendering is triggered first and the other but I don't know, you could just create a window that, let's say, in open layers you click something and then you get a timeline with a pop-up or some
24:41
and this should work more or less out of the box you just have to style the container you put it in that should be it alright okay is there no one else?
25:05
okay, good then, yeah, give applause to the presenters thanks thank you