Using GRASS GIS in Jupyter Notebooks: An Introduction to grass.jupyter
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69142 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 2022102 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Grass (card game)User interfaceGraphical user interfaceSoftwareOpen sourceAdditionCodeFunction (mathematics)Letterpress printingNichtlineares GleichungssystemLaptopSource codeAreaElectronic visual displayLevel (video gaming)Overlay-NetzModul <Datentyp>Raster graphicsDisintegrationProcess (computing)TopologyVector spacePattern recognitionDatabaseData managementTemporal logicAlgebraStatisticsInteractive televisionCircleVisualization (computer graphics)File formatComputer-generated imageryTesselationImplementationElectric currentArtistic renderingGrass (card game)INTEGRALRoutingElement (mathematics)Different (Kate Ryan album)LaptopKeyboard shortcutInstance (computer science)CodeFunction (mathematics)TelecommunicationCellular automatonLink (knot theory)Parameter (computer programming)Integrated development environmentRaster graphicsRepresentational state transferMaterialization (paranormal)Vector spaceSoftware repositoryGame controllerVirtual machineStrategy gameComputerElectronic visual displayMedical imagingStudent's t-testLine (geometry)Denial-of-service attackData managementFile formatSocial classInteractive televisionModule (mathematics)Block (periodic table)Series (mathematics)Open sourceRight angleMultiplication signSinguläres IntegralUser interfaceFamilyEndliche ModelltheorieScaling (geometry)Analytic setWeb pageLevel (video gaming)VolumenvisualisierungResultantVariable (mathematics)Visualization (computer graphics)DataflowGraphical user interfaceWeb 2.0Sheaf (mathematics)MappingOperating systemSpacetimeSimulationStandard deviationProcess (computing)Decision theoryContent (media)Artificial lifeParallel portLibrary (computing)Set (mathematics)Projective planeTable (information)Overlay-NetzComputer fileSampling (statistics)Order (biology)Revision controlFront and back endsCloud computingPower (physics)Selectivity (electronic)Lebesgue integrationMetreProjektiver RaumTime seriesSpring (hydrology)Portable communications deviceBound stateGUI widgetQuicksortTesselationGeometryLocal ringCuboidSoftwareNichtlineares GleichungssystemSingle-precision floating-point formatScripting languageStreaming mediaMathematical analysisReal-time operating systemSlide ruleSoftware bugGoodness of fitSource codeUniform resource locatorPresentation of a groupFrame problemDistribution (mathematics)Computer musicCompilation albumPoint (geometry)DatabaseExecution unitMixed realityDirection (geometry)Computer animation
Transcript: English(auto-generated)
00:06
Good afternoon. So we have in this session three talks, starting with Caitlin Heydrich. Caitlin is a PhD student at Center for Geospatial Analytics at North Carolina State University.
00:24
Thank you. Thank you all for sticking around to this late afternoon session. I'm going to talk today about GRASS and Jupiter, an introduction to GRASS Jupiter, and thank you to Anna and Vashek, who are also integral to this work.
00:41
But before we get started, yeah, I am a PhD student at North Carolina State University, which has another Geo4All lab there. I work with Anna and Vashek, advised by Helena Matasheva. And through that work, I'm also a TA for GIS 714, and I thought since this was an
01:00
educational theme of session, I would frame our discussion of GRASS Jupiter around using it in a course. So this past spring, I TAed Geospatial Computation and Simulation, which is a course that all of our first-year PhD students in geospatial analytics take.
01:22
And this is not a new course. It's been taught for many years using these web-based tutorials. So the instructions are online, and then students do it locally on their own machine using all FOSS packages, mostly GRASS GIS. And we decided that we were going to update it so that this year it was all based in
01:44
Jupyter Notebooks but still used GRASS. But before I dive into what that looks like, I just want to back up and make sure we're all on the same page about GRASS GIS and Jupyter Notebooks. So GRASS GIS is an open-source GIS software, geoprocessing software, and it has over
02:04
500 tools in its base installation, and then the add-ons have another 400 tools. So it's a huge toolbox that's really powerful if you can access it. And there are many different ways that you can access it. There's the graphical user interface, which I've shown the new single-pane window here,
02:22
which was released recently. There's also a command-line interface, a Python API, C, and it's also integrated with R, QGIS, and there's even a REST API actinia. And then Jupyter Notebooks are like a new and extremely popular format.
02:43
They're a document that makes markdown, like narrative text and even equations, with live code and then the output of that code. So in the background, they're running a... It's got like a IPython kernel, but you can also set it up with over 100 different other kernels.
03:00
So they're really powerful communication tools. I actually made this PowerPoint in a Jupyter Notebook, so I can embed cells and then see the output of them. So you can see, this is an example where we have that narrative text, the live code, and then the output of it, and the output can even be interactive.
03:21
When I turned it into a slide, though, this would be changing with this in a real notebook. Okay. So you can see that there'd be a big motive for us to integrate the two, because Jupyter is such a powerful communication tool, and it's really nice the way that it mixes all
03:44
those different elements, learning elements. And Grass does have this existing API, Python API, but there was just like a couple hurdles that would make the integration a lot better, like visualization techniques and some data
04:02
management strategies. So we developed GrassJupyter, which is a sub-package of Grass GIS. So when you install Grass, you're installing GrassJupyter with it. It was included in the main Grass GIS distribution, starting with version 8.2, and it's meant
04:21
to fit seamlessly like the syntax is consistent with Grass's existing syntax. And you can try it out in Binder. At the end, I'll give a link to this presentation, and you can click on it. If you aren't familiar with Binder, it's a cloud computational environment, so you can
04:42
try GrassJupyter out without installing it on your own machine. Okay. So we redesigned this course so that it used GrassJupyter. Here's the link to the course repo on GitHub, and you can launch it in Binder and try
05:02
all the materials that we used in this course last semester. And I think to show what GrassJupyter really looks like in practice, we'll just walk through a small excerpt from one of our notebooks that was on surface water processes.
05:20
Okay. So getting started with Grass and Jupyter. After you've installed Grass and Jupyter on your local machine, in the notebook, you have to start a Grass session, and so this is a pretty standard block of code that's at the beginning of all of our notebooks. We're pointing Python to where our Grass Python packages are, GrassScript, which is
05:43
one of the Python APIs, the other one being PyGrass, and GrassJupyter. And then we start the session with GrassJupyter.init, and that starts the session and it also handles some of the environmental variables in the backend that you used to have to set manually
06:02
if you were trying to use a notebook. And now we have a Grass session going. We can access all of our data. In this example, we've set our project space to the North Carolina sample database, and then our map set is our homework three.
06:22
Okay. So now that we've started our Grass session, we want to start doing some analysis. So we, in this first section, are going to model a small flood, like a lake. So this would be like we put a dam in or we're filling a bathtub. We give it a seed point, and then we can display the output with our first GrassJupyter
06:44
visualization technique, which is the map module. And so we create an instance of it, and then we add the two rasters, our elevation raster and the output of the previous command, the R lake, the flood one, and then we show
07:04
it with dot show. And this output is just a simple PNG image. And so it's pretty, it's our most simple visualization technique. And then because this was a course and it's really neat to be able to integrate this tutorial
07:23
material with the live code and questions, we ask students to give it a try themselves. And we can also visualize things in 3D. So this is the next GrassJupyter visualization tool in your toolkit. It's Map3D.
07:40
It lets you make 3D images. They're just PNG images. They're not interactive. But they, yeah, can make nice renderings like this, and you can save them on your local machine with dot save, and you can overlay all sorts of other things, like we
08:02
are overlaying a legend here with the dot overlay d legend. But you could also overlay like a bar, a scale bar. So the syntax of both these modules, I want to explain how it's similar to the existing
08:21
Grass display library. So if you're a Grass user, you know that Grass has a whole family of display modules. So in that toolbox of 500 tools, there's a portion of them that are specifically for adding graphical elements to the active graphics frame.
08:40
And so we can call those modules using this shortcut. We take the name of the module, which would be like d.Rast or d.Legend, and we replace the dot with an underscore and append it to the end of the instance of the class,
09:02
and it will route the contents of the parentheses to the module. So it is really convenient because it prevents us from having to reinvent the entire wheel. Instead you know that you can call any Grass family display module and send the contents
09:21
over that way. Okay. So our example moves along. We move from just making like this simple bathtub or dam simulation to modeling a flood along a section of river using the height above nearest drainage methodology.
09:41
And so it's nice because we can include all this narrative text. Here we are doing some watershed modeling, deriving the flow accumulation, the streams, the drainage direction, and then we'll go through all these modeling steps.
10:01
I won't go through it all. And then at the end we end up with this time series of the flood happening. And the output of the previous step, this R Lake series where we're making that flood, that inundation flood along the river, is a Grass space time dataset, and it's really
10:24
nice in the graphical user interface the way that you can make animations out of space time datasets. And so you can now do the same thing in a Jupyter notebook with Grass Jupyter time series map. And so similar to that syntax I was talking before, you can add base elements with D
10:46
underscore RAST or any of the other display family of modules, and then add your space time dataset with add raster series or add vector series, and then the output looks like this, and when I converted this notebook to HTML, the interactivity also died.
11:04
So I made a GIF of it. So now you can see the flood occurring, and it would be in line in the notebook, and it would have the sliding bar would be working so you could explore the results of your flood right in line in the notebook.
11:27
So we have map, we have map 3D. We have support for time series, but interactivity is a really handy tool when you're doing geospatial analysis, being able to zoom in and zoom out, toggle between layers.
11:43
So we decided to integrate Grass with Folium for that. So Folium is a popular Python library for creating leaflet maps, and so it creates these HTML maps that can be displayed in line, and it has a huge selection of base maps and you can even give it custom tile sets to put in the background, and you can export
12:05
your leaflet map and embed it on your website. So they're really convenient sort of portable outputs. But the decision to integrate with Folium came with its own difficulty, and so briefly I just want to touch on why it was difficult to integrate these two.
12:23
Folium, along with many of the other HTML mapping libraries, is in Web Mercator, and you might have a totally different projection of your data in Grass. And so in order to put your data in Folium, we have to re-project it.
12:42
And so that was one of the bigger hurdles in developing this library was figuring out how to re-project it to a temporary location, save it as temporary files that are in the correct projection and correct format to be consumed by Folium.
13:00
But what's nice about it is that you can pretty simply make these, like, really nice maps that display in line, so you can have the elevation map is from Grass. I'll go back to that. This is how I'm making that map, and the third block of text, GrassJupiter.raster,
13:28
is where we're taking the raster from Grass, writing it to a PNG, and getting the bounding box, and then adding it to the existing Folium instance.
13:42
And that makes this map. But this might be really difficult if you aren't a Folium user, to have all that text and learn all the Folium things, all that code. So we also developed interactive map, which is just a shortcut to doing something like that. This also broke when I was exporting it, but this map would look the same as the
14:02
other one. We are just wrapping it all into interactive map. The downside of interactive map is that you have less control over the Folium map, but it's much faster, a faster way of exploring data. Oh, and it lets you, like, toggle between layers if we had multiple layers there.
14:23
Okay, so we implemented a whole course that used GrassJupiter, and we had a bunch of different units, and we hosted all these notebooks in GitHub, and then we let students either run them locally or run them through Binder, that cloud computing environment that I mentioned earlier.
14:42
And it actually went pretty well. Students noted that they really appreciated being able to change the parameters in the Grass module, so you can imagine that if you were doing that flood model I was talking about earlier, you could change, maybe you want to have the flood go to six meters or two meters, and you can see that real-time right in the notebook.
15:01
And then having it hosted on GitHub and Binder was also really convenient, easy to access the notebooks. If students had trouble setting it up locally, they could run the whole course in Binder, and so we could sidestep a lot of technical issues that way. But we also did run into some challenges, there have to be.
15:23
For setting up the notebooks to run on local machines was difficult for many students, and even students that did get them running locally often encountered issues that were like Grass, specific to that operating system or their version of Grass, finding other Grass bugs, and occasionally Binder was unreliable.
15:43
We found that at times during the spring the computational nodes of Binder were all taken up and none of us could get the notebooks running. Yeah, so in summary, Grass Jupyter provides session and data management.
16:04
These four different visualization techniques, and it was really effective for us to implement in the classroom and to use it to explain a lot of high-level Grass workflows, and there's still a lot of work to be done in the future.
16:22
Yeah, implementing other mapping. Instead of folium, you could do IPIE leaflet or GeoViews, and I'm pretty interested in pursuing some of those, using different back ends for that interactive mapping, and I think that's low-hanging fruit because it's already a GeoJSON and a PNG image with bounds.
16:43
And then some parallelization and maybe improving the formatting of text output from some of the Grass modules so that it displays nicely in a Jupyter table, or Pandas table or something. Yeah, I'll come back to this slide, but that's the end of my talk, and I really want to say thank you to Google Summer of Code
17:02
that sponsored this work last summer, to the Grass community, which sponsored my mini project, and to Helena, Stefan, and Vero back there for all of their guidance, and to the class who were my guinea pigs. And if you'd like to learn more about this, to try it out, here's the link to the slides, and you can click on any of those Binder links.
17:23
They'll take you to that cloud-competing environment. You can test it out. There's also a whole host of notebooks where you can explore this on your own. We've got the manual page. There was a workshop on Monday that Ana led that is just an introduction to Grass more generally,
17:43
but in a Jupyter notebook, so it's a really great starting point. There's our course materials, and then now that we've integrated, or now that Grass is easier to use in Jupyter, on Grass' GitHub page, there's actually a bunch of notebooks that are also an introduction to Grass,
18:03
and so those are linked in the main Grass readme file, so you can find them there, or at this link. All right, thank you.