Building the modern GIS Web Stack
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68436 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Streaming mediaStack (abstract data type)Front and back endsDatabasePoint cloudCustomer relationship managementBuildingStatement (computer science)Covering spacePersonal digital assistantVariety (linguistics)Data storage devicePrice indexWrapper (data mining)Server (computing)Computer configurationMachine learningData modelFocus (optics)ModemContext awarenessLibrary (computing)Complex (psychology)Open setScale (map)Visualization (computer graphics)Euclidean vectorComputer-assisted translationWebsiteStack (abstract data type)BitWeb 2.0Mathematical singularityServer (computing)Set (mathematics)Front and back endsCASE <Informatik>Point cloudRight angleDebuggerLatent heatMereologyInstance (computer science)Mobile appLevel (video gaming)Connectivity (graph theory)Data storage deviceProcess (computing)Operator (mathematics)Client (computing)DatabaseInternetworkingNeuroinformatikMobile WebProjective planeAnalytic setFile formatExterior algebraNumberBridging (networking)Computer configurationLibrary (computing)Physical systemMultiplication signSoftware developerArtistic renderingFocus (optics)Entire functionVector spaceService (economics)2 (number)Object (grammar)Integrated development environmentRevision controlCore dumpContext awarenessTemplate (C++)CodeAbstractionScripting languageCartesian coordinate systemVariety (linguistics)Functional (mathematics)Point (geometry)GeometryChemical equationSmoothingQuery languageObject-relational mappingGene clusterStatement (computer science)Online helpCustomer relationship managementFluid staticsInteractive televisionElasticity (physics)Internet service providerTesselationType theoryGame controllerView (database)Affine spaceMechanism designExtension (kinesiology)Self-organizationBuildingCache (computing)Computer-assisted translationQuicksortBlogData conversionHuman migrationAxiom of choiceEndliche ModelltheorieDependent and independent variablesFeedbackData managementTwitterArithmetic progressionFormal languageWeb-DesignerGraph (mathematics)Sinc functionShape (magazine)Subject indexingOpen setPolygonStandard deviationPresentation of a groupSheaf (mathematics)Category of beingPattern languageElement (mathematics)Canadian Mathematical SocietyVolumenvisualisierungDenial-of-service attackState of matterExecution unitIndependence (probability theory)Reading (process)Computer fileRaster graphicsLecture/ConferenceComputer animationXMLMeeting/Interview
Transcript: English(auto-generated)
00:00
Building the modern GIS web stack. So we'll talk about that and we're here at Tartu, but first a little bit about me So my name is Jashan P. Singh, but I do go by Jason Singh. Also my website is jasonsingh.com Because at the start of my career Jason is what I was doing like I would keep on going to my colleagues Jason this Jason that and they would like
00:21
Send the Jason in this format. I'm a freelancer based out of India. I've done a lot of full stack development I started off with building front ends, then I moved to back ends and the databases in cloud On the right are my two colleagues that I work with Their cats one is neko one is kato Both of them are actually named cats neko is cat in Japanese and gato is cat in Spanish. So fun bit
00:45
Okay, defining the problem statement itself. What exactly are gonna be we're gonna be talking about So what exactly are gonna be talking about so building the GIS data management system
01:02
So as a freelancer and also like a web developer before coming into the GIS system Almost every systems can be generalized into building a data management system Which is essentially like you have some data either It's user generated or some something else generated and users are interacting with that data Right when you and when you add the GIS part
01:23
The only thing that it changes is now there is a spatial component to it But again, that's like a huge property of a GIS management system system But a lot of the concepts of data management systems also plays a part while you're working with GIS data It's not separate both the industries coming together while from the yes side and from a standard CMS or data management side
01:46
So the problem statement What like what I'm gonna be talking about is building such how to build such a system What are the different concepts involved how you can tackle them and like gives a sort of a template on how you can go? about it the I mean, of course and this is like a generalized way of going about it and
02:06
Like it's not gonna cover every use case So the idea is just to cover a wide variety of use case it's fast to build and easy to manage right, so any System has like these different parts. There is front end where the users see and interact with the data
02:20
There's a back end where you process the data There's a database where you store the data and cloud which is essentially running it on someone else's computer or more Appropriately putting it on the internet. So That's what these are the different components that we're gonna be talking about Right, right. So these are different topics that will go through. What is modern? Why Python? Why fast API?
02:43
What's gonna be the front end where it's gonna be hosted? What's the database? What about the cloud all these different aspects of? Building a GIS data management system, right? So what exactly is modern? I mean, it's it's very hard to define. What is modern? It's very subjective
03:02
It's very context specific for somebody modern could be very much Going from old service to a new service, but that's also it might still be old but modern in Modern in today's world and how I would define it that you build it for the web You build it so that it's built for accessibility. It's easier to access. It's interoperable and
03:25
Everybody is able to work with that data so the focus of this talk is to make it more web oriented and have some cloud native support and Use like the modern tooling that is available to work with that data Right, so the stack
03:41
That I propose or the stack that I've worked that stack that I've also Gone through while working with a lot of GIS data management system like all the different Projects that came to me. This is like the most generic or the most Common stack that I worked with and also found it to be like able to solve majority of these problems
04:03
The back end is Python plus fast API fast API is relatively new Front end is in react plus map library react It's old but still like changing every time and map library is relatively new Databases Postgres post.js even though post.js is not new it has been
04:20
Able to tackle any sort of problems that you can deal That you can that you need to deal with in your GS database or GS data sets and cloud In today's world we can run on any cloud We don't need to stick to one cloud stick to their services So for the cloud, it's just basically any any I've not faced any issues while migrating from one cloud to the another
04:44
It has been very easy that way So let's talk about the front end. So for front end a lot of As a freelancer I've approached by clients and a lot of the times there They are heavily focused on the front end. Like they don't generally never even talk about what's gonna go into the back end
05:04
It's majorly front end. Like the front end has to look good. It has to look smooth and This also one of the first things that they talk about you get we need the front end to be ready We need the front end POC to be ready. We don't care about what goes into the back end, right? So for front end, I mean you have a lot of options react is just one of them
05:25
I found it to be highly versatile. It can cover a lot of use cases and Again, this is not react doesn't have anything to do with GIS, right? this has all to do with building a UI building a better UI a smooth UI UX and
05:42
It's still one of the most popular front-end libraries I mean others are getting there but still even if you are if you are going out hiring for talent You will still find there are more react developers Right, and there's a huge component support like depending on what design system you choose You'll find a lot of compatibility with react ecosystem
06:00
Right, so that's about the core React component the map engine the GIS part now that is I've chosen to be what I've chosen it to be a map libre it's a growing community and Gets everything done by that I mean mostly I mean not everything everything if you're still looking for a very high level of 3d support
06:20
you still have to go to cesium and Alternatives like that are listed down mapbox and leaflet leaflet is still a very great library for doing a lot of stuff But again, it doesn't support it doesn't have web GL support so if you are if you if you're going to work with a lot of Heavy or like if you with a lot of data and you care about performance
06:41
Then you have to stick to something that supports web GL. So Map libre will be there and open layers open layers is actually quite well There's also a stack support for open layer and map libre doesn't support stack out of the box as of now You know depends on your use case if you have like a stack data sets and you want to get things done and you want to
07:02
visualize a cog so open layers would be your choice and Then additionally, there's also deck GL. It's not part of the map engine, but you can also use it to To load a very huge amount of data in a very in a relatively much less seconds
07:20
It offloads all of your rendering and all of your data processing to the GPU Michael II here she gave a good workshop on it. So if you don't know So moving on to the back end So the back end the choice was quite I mean, it's usually very simple There is there's always a Python packages. There's always a Python package that can get the stuff done
07:45
You need to do with the geospatial data So it made sense to have Python as a back end a lot of a lot of times I've met a lot of people and I've talked to a lot of people a lot of people are doing just Python scripting in QGIS. So Python is the language of choice for a lot of people who are working with the geospatial data, right?
08:04
So Python that's it. But on top of that there is also like it's easy to use I mean, it's very intuitive as a language and there's a huge community around Supporting and adding more functionalities to the geospatial side of Python now fast API. I haven't
08:21
Haven't mentioned I haven't heard it talking people a lot about fast API. It's it's relatively new, but it's not that new initially when when you're working with Python on the web you had More or less two choices other it's flask or Django now flask It was too bare-bone for you to get anything done
08:40
But Django brought in a lot of concepts of their own and how the data should be structured it worked Django was initially built for Journal website or journal company to build a lot of a lot of blogs and work with a lot of blogs It does have a lot of it does have support for geospatial functionalities and geospatial data
09:02
But it also forces you to think in a Django way fast API Brings in like a good balance between the two you don't you're not bare-bone like flask and you're not overly Overwhelmed by the Django concepts, so it's something in the middle and It really helps you get you
09:22
Get you an environment to start working on the API part and not focus too much on The web part like lot of it is handled by fast API and you just focus lot on working with your data, right? we'll get into a little bit detail of that and Finally the database, right?
09:42
So you the users can see you have figured out where the users how the user gonna see the data interact with the data We're gonna process the data not talk about storing the data itself now Post GIS, even though nothing new but has been able to handle almost anything and everything
10:00
I've thrown at it and it's still an active development and the number of extensions keep on growing I didn't even knew something like mobility DB existed till somebody came to me with a project for Building a taxi Analytics app and it just works like a lot of lot of geospatial ruta and you can do in Postgres and you don't need
10:23
Anything beyond it right and for raster data sets. I mean for rasters Postgres Yeah, I mean you can but still it's not very recommended way of storing your rasters and working with the rasters Well, so for that object storage is still the best choice. Just put it on a bucket s3 bucket
10:41
You should be bucket with a node bucket. It doesn't matter Make it make them cogs and even for even if you have static vector vector files You can use PM tiles and you have like a built-in cloud native supports and you don't even need a server to get The data displayed on the front end So That's what I've been working with and there's also like very amazing library uber-h3. I
11:04
Last possible G I had to talk about how you can model your data in extra indexes and use the fastest shape spatial joints for for building very high performance spatial analytics and there's also a extension for stack pg stack
11:21
So the cloud for cloud, I mean there is really no Good or bad. It will usually it has usually come to an organization Some organizations have affinity for AWS some for Azure or some for on-prem Also, it it's really it's really very open-ended and never seen like
11:43
Prop a for sure affinity but I've all but every but docker makes it really easy for packaging your back-end applications if you have docker and In in the previous talks and conversations, I've also heard a lot of people using kubernetes for working with the geospatial
12:02
Applications and clusters. So docker is just in like an entry point into that whole ecosystem of cluster cluster management and all of that So package your application as docker and you'll you'll never have to worry about which cloud you're running in for postgres you can run it in a VM or just all the user
12:21
docker instance, right and service it all works But it gets really expensive really fast with the amount of data that you have to do And especially in context of geospatial data that data grows pretty fast I've seen databases for up to 2 TB 3 TB 4 TB and for
12:40
Storage like that. You have to pay a lot of money to your service providers So it's better to run it in a VM or have your own cluster management Static storage for front-end That's just like if so This is a more of a concept of whether you're gonna be running your front-end in a server side Components or whether you're gonna be running it on the cloud. I would recommend that you run it on a
13:02
client-side Because anyway, you won't be able to do any of the map stuff on the server the map The canvas element will still load on the client-side So a lot of the performance hit is gonna be that so there is no point of running it on the server Just use a static storage. You will have better Cache control and the website will be quite fast and any s3 compatible storage for your
13:25
Cogs and for your tips and any of the last two data sets that you have even the pre render tiles that will work right But again, this is not just something that you can do. I Mean, it's not the end of the world
13:41
Given all of this stack It still doesn't make sense sometimes to not use it and sometimes use part of The applications not built in this I found geo server to be a great for all your back and stuff like it can connect your post Yes, it can connect your static server And if you just want a data Backend to just have to just load the data and you are focused more on the front end
14:01
Then you can just use your server on the back end. Teria JS is kind of new I've not heard a lot of people talking about it. It's also from the Oceana side of the world so it's also if you have a lot of cesium If you have a lot of 3d data that you are working with so Teria JS is great to begin with don't directly jump
14:21
On to cesium. It's very difficult to work with unless until you like like absolutely a lot of challenges so Teria JS is good to work with if you have like a lot of 3d support map store and Apache Sedona still I haven't worked with them But I've heard like good things about like how people are using it So these are just mentioning alternative stacks
14:42
if we have some time so I Mean enough talk. Where's the code? So I've been building a GIS template for as a beginner for people to you know, get started with the whole GIS Full stack application. It's still a work in progress I've just done the bare-bone front and stuff and working on the back end and
15:02
I've listed down a couple of features. So feel free to create issues if you can contribute great But if you can't just send some feedback, that will be great So deeper into the code house this this section I put it only if I have some time left and since we have some time left So we're gonna go over this so deeper into the code
15:21
front-end The core part as I said was react for components I mean, there's a huge number of libraries if you want to support like material style, then there is material But I would recommend shaazi and it's a very Very great library for building a lot of stack Yeah for building your components and you can only pick the components that you need and not the entire thing
15:46
For stylings, even if you have heard bad things about tailwind tailwind, it's pretty good You don't have to worry and also like this what why I'm saying all of this is that a lot of people here Are just more focused on GIS and not the web aspect. So I'm simplifying a lot of these things the use tailwind
16:04
And you will don't have to worry about a lot of CSS stuff For mapping units map Libre. It's a growing community If you put it on the slack channel Somebody will help you with all the stuff that you need for state management in react You can use Jota for API client You can use just basic fetch and for API cache you can use react query now this last point
16:23
It's very important if you have a lot of GIS data and you're making a lot of requests to the back end Don't DDoS your own back end, right? Don't keep on sending the same request and getting the same data from the back end So it's really really important that you have some sort of cache mechanism on your front end it will make your website really smooth and
16:45
It will like it and people and users will have a lot of and users have a good time interacting with it Fetch your request. Sorry cache your request fetch your request and then cache your request and don't invalidate that cache Till it's Till it should be
17:01
For the back end going deeper into like what the back ends would consist of for the pipe So the core is again Python and fast API models is pedantic pedantic is used to Declare what the response and the request is going to be look like this is really important for People working in large teams and if you have if you want to have a common understanding of what the API will look like
17:27
Database migrations is Alembic. So If you have a database and a lot of people are working on it or even it's just like two developers working on it You have to have to create a version control system of your database how it expands how it grows
17:41
So Alembic is where you write your migrations and you run the migration and see that everything is intact, right? So package it in docker use ORM. I would highly recommend that you use ORM and not write your SQL right off the bat But if you're feeling risky then sure for GIS libraries, there is also rasterio and GDAL
18:01
I'm sure a lot of people have used it already for their Python scripts We print the code cloud. So Similar concepts apply on the back on the cloud itself If you are writing your code and your version maintaining your control, it makes sense to version control your cloud as well So for that one of the two popular tools are terraform and ansible
18:22
But if you are in a specific cloud, for example AWS, then they have cloud formation templates for working with versions or services so you can also use that azure has ABMs and On the cloud use cogs and PMTiles as much as possible. I mean those are great formats reduce your
18:43
Reduce your data fetches by a lot You can you only get the data that you want and Postgres as I said, you can done it in docker and VM But if you are if you are working with a lot of data and you need to have a cluster management system There is a great tool by Crunchy Bridge called Postgres operator. There's also the same by Percona and all of that so you can run it in kubernetes
19:05
the whole Postgres cluster and Shift the processing on the main instance and just to the read from the secondary instance Okay, so Focusing on the key takeaways. There is no server bullet Obviously, this is just a very opinionated take on what your full stack would look like
19:24
Right and but you can always bet on Python and post here has to handle everything that you need focus on the feature that you want it should be faster to build and easier to manage and You have to have some cloud native support built into it for better performance and lower cloud costs Right and thank you. And that's me
19:48
Thank You Jason So we have time for a few questions Please raise your hand and wait for the microphone as usual
20:06
Hey Thanks for presentation. Actually, I use also this quite similar stack for my own also project, but How you suggest the stack deals with not only rendering the data from databases
20:22
But when you need to put the data from the client side to the database Did you use some Tools like or just from the fast DPI you built Like custom API to update data from client side when they are editing data or something like that Are you talking about like when customers build polygons and they?
20:44
Polygons or if your customer is loading a TIF and then you upload that onto the server stuff like that. Yeah Yeah, it's currently it's very custom But to work with a lot of stack and cogs and also a lot of vectors in your data there are T Tyler and tip PG and those either you can run them as an independent instance
21:04
I did mention it in the slides but either you can run them in the Independent instance or integrate them into the fast API and use those API's and whatever API's you want You don't have to like bring in the whole ecosystem also Because I also building like custom way and always thinking about maybe I'm
21:21
Wrong. No, you know this maybe You know You should follow the T Tyler and tip G example look at their code and maybe build a library for fast API that people can Directly integrate into loading the geospatial data. So that would be a great way to give back to the community. Yeah Hi, you said that cesium was a tough thing to start with and I was wondering does map Libra have
21:47
any 3d Capability or just some primitive stuff for some 3d they To phrase a map Libra they call it 2.5d. You have the hillshade and terrain stuff. So You can visualize some of it
22:02
But if you're working with a lot of 3d data like buildings data like LOD 1 2 level 3 like up to 4 you won't be able to do that in map Libra and Map Libra community is also looking for people to you know, build that support So if you are really interested you can go and create a PR and have that much
22:20
What map mentions to you such as for 3d? cesium
22:41
Yeah, I mean there There is really no better alternative I have yeah, I The the library that I mentioned Teriages so they it abstracts a lot of the cesium stuff For you to so that you don't have to directly integrate you don't have to directly work with cesium And you can do a lot of that stuff
23:02
But really yeah cesium is the only way Fortunately, sorry, unfortunately more questions I'm curious to hear you a few on a number of things that are not here like
23:23
full text search elastic search so all are these type of things and maybe graphile as an API Like so about with graph QL But still like in the lot of geospatial context, it's still like a lot of data It's just happening on the rest API side of things. I mean, it's it again
23:43
It's very so this was supposed to be a very generic and easy to do For if you introduce if I introduce your graph QL concept Then you have to understand what graph QL is you have to manage it on the front end and on the back end since a lot of Technologies like even stack they have a rest API client So a lot of this industry strength inside GIS industry is still like on the rest side
24:06
So it's I'm still kept it like you stick to rest for now And then maybe if you want to you can use build a graphical client as well and regarding open search and elastic search and Kibana and all this stuff this again goes into
24:22
Kind of high-level stuff when you when you have like a built-in when you have the entire system and you want to understand different patterns what people are doing what people are not doing how they are interacting with the data how the data is Being stored managed and what's the performance metric? So it's slightly high level We have time for one last quick question. This is quite the broad
24:46
Topic You you are always tired. There's me I guess So, let's close the session here, let me start by thanking Jason and the other speakers Thank the volunteers, especially IO and think also the technical team, which I think it's only one person
25:05
But this is only going going very smoothly So thank you and thank you to the audience for being here and interacting with the speakers. So enjoy your evening