QGIS Server in an enterprise environment
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68526 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu106 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Software developerSoftwareOperator (mathematics)Enterprise architectureWeb serviceComputer networkAutomatonComputing platformSoftware testingPlastikkarteTransformation (genetics)Configuration spaceServer (computing)Vector spaceComputer fileRaster graphicsComputer configurationVariable (mathematics)Integrated development environmentData storage deviceSheaf (mathematics)Witt algebraUniform resource locatorDatabaseIntegrated development environmentSoftware developerServer (computing)Enterprise architecturePresentation of a groupProjective planeVariable (mathematics)Computer fileProcess (computing)FrequencyLevel (video gaming)Strategy gameCache (computing)Product (business)Open setInterface (computing)Pairwise comparisonGeometryConfiguration spaceComputer configurationMereologyWeb 2.0Revision controlOperator (mathematics)InformationWebsiteComputer programmingDatabaseAttribute grammarState of matterMedical imagingSource codeCategory of beingProxy serverParameter (computer programming)AliasingWindowCartesian coordinate systemData storage deviceBitMultiplication signDomain namePoint (geometry)Uniqueness quantificationSet (mathematics)PlanningComputer-generated imageryAdditionGateway (telecommunications)Task (computing)Set (mathematics)Table (information)PlastikkarteConnected spaceContext awarenessTelecommunicationLatent heatInternetworkingMappingMultiplicationLecture/ConferenceComputer animation
10:03
Uniform resource locatorDatabaseServer (computing)Client (computing)SoftwareProxy serverInterface (computing)Process (computing)Enterprise architectureDependent and independent variablesFront and back endsInstallable File SystemSanitary sewerConfiguration spaceProbability density functionDirected setDisintegrationAutomationFile formatWeb serviceLogicBitUniform resource locatorServer (computing)InformationOpen sourceProjective planeConnected spaceTraffic reportingAutomationStandard deviationMoment (mathematics)Configuration spaceNeuroinformatikSlide ruleSystem administratorIntegrated development environmentUser interfaceDatabaseDifferent (Kate Ryan album)PasswordPoint (geometry)Computer fileDependent and independent variablesFront and back endsEmailSoftwareWeb 2.0Medical imagingFeedbackCASE <Informatik>Client (computing)Plug-in (computing)Domain nameLink (knot theory)Formal languageGeometryReverse engineeringInformation securityKey (cryptography)Cache (computing)WordInstance (computer science)Repository (publishing)Proxy serverInterface (computing)Level (video gaming)AuthenticationLetterpress printingProcess (computing)Probability density functionArithmetic progressionFigurate numberRewritingComputer animationLecture/Conference
19:55
Instance (computer science)BitServer (computing)TesselationCache (computing)Term (mathematics)Pairwise comparisonThermodynamisches SystemSet (mathematics)Computer filePosition operatorDifferent (Kate Ryan album)Projective planeMultiplication signComputer programmingArtistic renderingMultiplicationPoint (geometry)Open setUniqueness quantificationDatabaseDimensional analysisRevision controlLastteilungClient (computing)Similarity (geometry)GeometryEntire functionLevel (video gaming)Point cloudPresentation of a groupWeb 2.0Computer animationLecture/ConferenceMeeting/Interview
25:19
GEDCOMComputer-assisted translationComputer animation
Transcript: English(auto-generated)
00:00
Thanks for joining the presentation. I will talk about QG, a server in an enterprise environment. So first, short about me. My name is Jacob Micksch. I'm a software developer at the company CityCom. You can find me at various places in the internet.
00:21
Our company is doing telecommunication solutions among many other things. I'm part of the smart infrastructure and broadband department, especially for planning optical fiber. In this context, we are developing an internal spatial data infrastructure and we are evaluating a couple of mapping servers.
00:49
We finally try to use QG as a server and I would like to present you what we have learned, what we use it for, what our current challenges are.
01:02
Also, I want to give you an introduction what it is and how to use it and install it. First, I will start with a rough overview, then give you some ideas how to configure it, how to install it, show you an example project. Then I want to especially focus on the connection between PostGIS and QG as a server.
01:27
Then how we use it in the company and then our plans for the future and final conclusion. QGIS server is a geodata server for WMS, WFS and WCS.
01:44
Also, OGC API features is already also in development but I haven't tried it yet. The cool thing is it can pretty much read everything what QGIS desktop can read and it renders the data exactly as QGIS desktop renders it.
02:05
This was the unique selling point why we decided to use QGIS server because we have our planners. They really have a lot of sophisticated styles for their domain specific tasks and we would like to have them exactly like them in the web GIS.
02:27
Because converting styles sometimes works but very often you lose something and that's why we are really happy that we can use the exact same rendering engine as QGIS desktop has. Then the configuration works exactly also in a QGIS project. I will show you in a minute where you can do this.
02:50
One thing which is important to know when you install your QGIS server is that it actually doesn't directly come with a server.
03:01
It's a fast CGI program which is a common gateway interface. So it's basically a program that runs locally and then you need an additional web server like Nginx or Apache to connect to this fast CGI interface and then you can publish it to the web so that other programs or websites can actually access it.
03:28
This is the part where you actually need to do some manual configurations here. Then a short comparison to GeoServer because this is also a very popular server.
03:44
Here I just have a picture like what GeoServer can read and what it can publish and I just striked through the things that QGIS server cannot do. You see it's still quite a lot of things and actually it can read and publish the things that most people use in my opinion.
04:05
It can be used for most applications but of course it always depends. Then how to install it. There is a manual where there is an instruction how to install it on Linux or Windows.
04:27
Of course you can build it from source directly or you can use pre-built options and then you still have to configure it with your own Nginx or Apache. There is of course a docker image which is our preferred way to install it.
04:46
There is actually multiple docker images. There is an official one which is relatively new. Then one from Oslandia, one from Camp2Camp, then QWC services, Cartosa and Hookla. The official one is the one we are currently using.
05:04
They state on the website it's not ready for production yet and that's why we would like to test it and make it ready for production at one day. So it also depends what you are looking for because all of them have some different flavors and how they are configured.
05:29
Can you actually configure a project itself or QGIS server itself? When you want to publish some layers you create a QGIS project, add some layers
05:45
to it and then it will be understand by QGIS server and can be published. You can store the QGIS project either as a file or in a PostGIS table and the QGIS server has to have access to this of course.
06:01
Also there are some global settings that you can give to a QGIS server like how many processes it should have or some really low level things. You can do this with environmental variables. So let's have a look how a project could look like.
06:23
So we have a very simple project with three layers here and if I would like to use it for QGIS server I go to project properties, QGIS server and then I have many tabs here where I can do some settings.
06:41
For example I have one for WMS and WFS and many more and the interesting thing is that all layers are directly published as WMS and you can also do some extra settings. So for example when you want to do a feature info request then normally only
07:06
the attributes come back but you can also set that the geometry should be returned. This is especially useful if you connect WebGIS to your QGIS server. And for the WFS you have explicitly to publish your layers so you just go to the WFS tab then you click on the layers you would like to publish.
07:32
And it's also possible actually to make some CRUD operations so you can even edit data via WFS-T so the transactional version and
07:46
then you can manually say what operation should be allowed and what should not be allowed but we currently don't use this feature yet. Also one very convenient thing is that you can go to the layer properties of your layer and for
08:07
every attribute you can create an alias because very often these attributes of your layer are not really intuitively understandable. But you can write an alias there and then the QGIS server will use this
08:21
alias to publish your data so it will be much easier to understand for external users. Then how to access a QGIS server. So this is a very simple request. So first of all I have here my endpoint then I say it should be I want to service WMS
08:42
and I want to have to get capabilities document so where everything is described what kind of WMS the server offers. And then it's very important that I have to specify which QGIS project I would like to have. So I have this map parameter and I have to point to the file the QGIS project file where the QGIS server should open.
09:10
But this is obviously not so beautiful and there's another option to actually put also the QGIS project inside a database.
09:21
And this is a bit nicer because then many people can access it at the same time and then there's no problem with multiple editing because the last person always wins. And this is very important if you want to store your QGIS projects in PostGIS and what you read it with QGIS server.
09:41
You have to use this environmental variable cache strategy periodic and then it will instantly update your services when the respective QGIS project has been updated. So but the problem is when you want to open a project which is stored in PostGIS it looks a bit more difficult.
10:06
So first of all you have to add the Postgres connection so if username, password and everything so this looks really ugly and also the normal user should not provide this. So there's one simplification that can be done. You can use so-called service files which I will explain in the next slide.
10:31
So then it will be simplified. You just have to show where the database is located and then you can just say here I have a service called my service and this comes in the next slide.
10:46
It's pretty much one file on your computer where you have stored a lot of Postgres connections which are so-called services and you can name them differently. So you can say my connection A, my connection B whatsoever. In my case I say it's called my service
11:07
and then I can write down all the credentials and this file is located somewhere on the QGIS server instance. And QGIS server has access to that. So and then it only needs to provide the
11:23
name of the service and does not need to provide the credentials directly in the URL. So then it looks like this. So let's say I have three different QGIS projects. One
11:40
is Sriga, one is Helsinki, one is Tartu. Then I can add this project name here. But we still found this is not nice enough. So we thought it's a bit of a challenge because the client, the web GIS or the users directly should have a really simple interface
12:02
and they should not actually deal with database credentials at all because of security concerns and so on. And so the solution was to build a reverse proxy between the client and QGIS server. In this case it's the Nginx server what we use here.
12:21
And so this figure shows like the basic process how the data is processed with QGIS server. So I will start very at the beginning. So here we have a user, for example a person typing using a web GIS doing a WMS
12:40
request. So it has the base URL, it says slash API, Tartu and then does a WMS request. Then we have a backend written in Go which does custom business logic and also authentication. This will add a header to tell QGIS server later on to adapt the links inside the capabilities
13:10
document that it looks for the client as the project would be served directly by the Go backend. I will come to this later in a minute. Then this WMS request will be forwarded to the Nginx which does a URL rewrite.
13:31
So here is the phrase Tartu will be taken and the URL will be rewritten or like written at all.
13:42
So it adds this PostgreSQL service and the project Tartu and here it references a service which is written in the PG service file. And this will be sent to QGIS server. QGIS server gets this request and sees okay they reference custom service.
14:04
I will look into my PG service file, finds the credentials and then can access the PostGIS database where the QGIS project is located and also all the layers. And then it sends back the geo data to QGIS server and this makes a WMS or WFS response and sends it back to Nginx and through the Go backend.
14:32
And again why we need this header? So let's say if this person makes a get capabilities request you get an XML document with all references to the different layers what you have.
14:48
And within this network QGIS has own domain or in our case we have a docker image which is called a QGIS server with a specific port.
15:02
And if you don't configure anything then the capabilities document just has the URLs written like QGIS server docker image, colon and then the port name and so on. So when you add this header this internal URL will be overwritten by the URL the user should see from outside.
15:27
So this is very important that you don't expose any internal information to the outside. And yeah so with Nginx you can actually really fine-tune how the URL should be rewritten.
15:42
So this kind of depends on your business logic what you have. So it's really powerful but I found it a bit complicated at the beginning but at one point you get into this. Alright then for future topics currently we have to configure all the QGIS projects manually.
16:07
So we have a lot of different projects and then I have to manually drag in the layers and publish them or not and so on. And it would be much nicer to do this in an automated way. And there's actually a new project called QSA so QGIS server administration which provides
16:25
an API to QGIS server so you can automatically configure your QGIS server projects. We haven't used this yet but we are trying to use it and also make some improvements if possible. Then also we would like to use the new OGC API standard so for example OGC API features.
16:51
We tried it already but there was some feature missing what we were needing. Then in the future we would also like to edit the geo data directly from outside.
17:00
But currently it works with WFST and you have to send XML documents through QGIS server which is not super nice. Especially nowadays when the whole web is working with JSON. So another thing is that we would like to directly integrate it in our desktop environment.
17:23
Because currently we only implement it in our web GIS. And QGIS server has also many more features which are interesting for example PDF printing. You can create really sophisticated reports in a QGIS desktop and those can be dynamically sent to the client.
17:43
So you can create reports, you can get it via PDF and you can do all the configuration in QGIS desktop so you can kind of configure both worlds. And also QGIS server we found some performance issues so you can of course handle them.
18:06
So we are working with some caching and so on and we are still trying to find out more how to do this in a better way. Then finally some conclusion.
18:21
So what is good about it, the styling is exactly like in QGIS desktop. And also the configuration can be done directly in the user interface which is also a bad thing which I will come to in a moment. And accessing the database can also be a bit complicated so we figured out how to do this with this PhD service files.
18:48
At the first moment it's a bit complicated but once you got it you find it's really elegant and a nice way to handle it but you need to get into it. The not so good side is that the automated configuration does not work yet but it's in progress.
19:04
And of course it does not support everything what for example GeoServer or MapServer does but this just depends on your project. And the installation also really depends on your use case so you still have to configure a lot by yourself.
19:21
Then I have some links for you and just QGIS related off-topic thing is that in our company we developed QGIS plug-in repository written in Go language. Because we have some internal QGIS plug-ins and that's the way how we distribute them through our colleagues.
19:45
It's open source you can check it out and it works in Docker environment. So if you would like to try and have feedback we are very happy about it. Good then thank you for your attention and I'm open for questions.
20:05
Thank you Jakob for your nice presentation. Question please. Do you have any experience on the performance if you compare it for example GeoServer or MapServer. When would you recommend to use a QGIS server for a client?
20:26
So the unique selling point is the styling that it's exactly looking like QGIS desktop. And we have like hundreds of different styles which are only written in QGIS. So that's why that's the only way we can publish them at all.
20:44
I have not done a performance comparison but I can imagine that maybe other servers are faster in terms of WMS. So this is the bottleneck the rendering of the WMS and of course you can tweak some stuff but I still have to figure out. But I would recommend if you only need to publish one or two layers then maybe GeoServer or MapServer are maybe faster.
21:17
Thanks for the presentation. Do you have any experience on how to
21:22
improve WMS services performance by putting it behind CDN or for example Cloudflare? Do you have some ideas about that? Because we are thinking about that and checking out maybe someone has already thought about this experience.
21:41
So what I'm currently trying is to use the program called Varnish. So it's a cache, it remembers the requests and I just made some time to remember for three minutes or so. That's why when the same tile is requested it will be just stored for three minutes in the cache and then it will not be re-rendered.
22:05
Also in our WebGIS we use the tiled version of WMS so it will always request the same dimensions of the tiles so then you can also cache them better.
22:23
Honestly I still need to figure out how the caching in QGIS server itself works. So currently we use Varnish and it works quite nicely. Can you maybe talk a little bit about the scaling? Let's say if we want to have multiple servers talking with the same QGIS project is it possible?
22:51
So you mean where are the project stores? So now basically I want to create a system where I have multiple, so for example we do this
23:01
with GeoServer where once the request comes we will decide from let's say the six GeoServer that we have. Depending upon the availability the request will go to a server and it will just stick there for the entire session. So can we do something similar? Actually I'm currently experimenting with this as well that in our Docker Compose setup I have multiple instances of QGIS server.
23:27
So then when you see in the log files you can see that different instances of QGIS server are called. I have not made a performance comparison yet but this is also one way how to increase the performance.
23:45
Yeah with the same project, yeah. Maybe just to answer that question also like just that works so you can essentially start QGIS server in like I don't know however many containers you want. If your database back and behind that can deal with that so because they look all into
24:03
the same database then you should be fine if you have a load balancer in front of that. And it might be a bit tricky in terms of caching so if you have a tile cache in between or something like that it depends a little bit on what you have around. But basically that works if you just like spin up a dozen of instances of QGIS server and if your database can deal with that then you're fine.
24:26
Otherwise if you build Docker containers in a way that you essentially put everything into the Docker container so that the container is basically self-contained like has no external dependencies then you should have no problem at all.
24:45
Just one more thing about this tiling. When you request a tile WMS you have to think about that for example labels are always on like could be cut off at tiles and when two bordering tiles have both cut off labels and they are not cut off at the right position.
25:06
You can have really strange artifacts and there's actually one kind of hidden setting in QGIS where you can force QGIS that the labels should always be completely inside of your tile then you don't get strange artifacts.