I just sent over to Joost Van den Broeke. He wants to tell us something about quality of service monitoring for OGC web services. So please.

Yeah. Thank you, Thomas. Yeah, I've been here for a long time. I've been here a long time ago. And then we're the first to speak in English. That's my German, my German, my German, my German. So I will continue in English.

This is about geo health check. Like Thomas said, it's a more officially quality of service monitor for geospatial web services. And we'll see what it is about. And like the last speaker, we're all using RevealUS. So I give some credits to the maker, Hakeem El-Hattab.

Thomas already told a little bit about me. I'm from the Netherlands and doing various things there. For one thing, I'm trying to pull OSG ONL as well, a local chapter of OSG O. And this project,

I'm not doing this by myself. Actually, Tom Kralidis started this project. Probably many of you know him. He's also working on various other projects. And Hannes Reuter from Germany. And he's also one of the contributors.

And there's many more, but these are main people. So here, probably, I don't have to explain what OGC is, what OWS is. I think the last speakers were already very detailed into the services. So I skipped this a bit.

So the actual contents of my talk will be, well, first, some monitoring challenges. And we'll go from there. We'll go from there. So suppose you're running OGC web service. You're running your WMS, WFS, TMS.

OK, not entirely OGC, TMS, WMTS. And at some point, you get a phone call from a user or an email. I see pink tiles. We have some German. Ich seh rose kachl.

He says, yeah. And this is what the user would expect. And this is what the user often would see. So here, you have your pink tiles. But those tiles are not sent by the server.

The server says, well, I'll send pink tiles today. No, actually, behind the scenes, there's a problem. And the problem could be manifold. It could be that you're not receiving an image, but you receive a well-formed exception report. Or even worse, you could receive the error

in the image, which is one of the WMS options. My point is, oh, yeah, the other thing. Let's say your database is being filled every night. And somehow, the process fails, and you get nice, white images,

but still valid images. My point is, in many of these infrastructures, regular HTTP monitors are being run. And as the OGC services tend to have their own error handling on top of regular HTTP, it would be noticed.

So 200 means everything's OK. You get the white image. You get your exception report. So your monitor, your regular uptime monitor, will never detect these situations I just saw. We go further. You can have a get capabilities response, which is OK,

well-formed. It could be a static file, but it will never guarantee that any of the more detailed servers will ever work. Recently, I've been doing some time-based services like sensor observation service, sensor things API,

irregularities. So for instance, we're working with 52 North. Probably everyone knows several people here in the conference. There's a source viewer. But you could have, let's say, a gap in your data. How would you know this? So this requires, all in all, very more detailed monitoring

than a generic monitoring. And the other thing is there are many uptime monitors you can register, like Pingdom or their uptime robots. I use those as well, because they send me an SMS as the service really down. But many of the critical OTC services,

let's say, when in governments, they run internally on internet. So you cannot use an external monitor. You need to have something internally. Usually, the IT department has something like a ping. But I hope I made my point a bit that we need an OTC-aware

service not just for uptime, but quality in service in general. And it would be nice to have history capture, because one of your users may call and say, well, the service is down. And you look, and now the service is up.

So you need history capture. So I hope I made this point clear that there is a need. Actually, I was looking for such a tool. And then I came along GeoHelcheck, which was already started in 2015. And I started contributing to the project.

And we have a demo running. You can register there and register for service. I'll give a short walk-through. So there's still lots of work to be done on the GUI. But the main thing is it has a dashboard where

you can register your services. And I will show you can also register the specific checks that you want to do. So here you see already some statistics. And you can zoom in on a specific service. You can see it has been offline, or it had errors.

And then it got back up. So the scenario is really you can log in. And you can enable if people can register, or you can disable that. And then you can choose from a list of services

that we support currently. It's an extensible thing. We recently added, it's not yet here, for instance, GeoNode, but here you would see a list of most of the regular OTC services.

So you can add what is called a resource. And then you give just the endpoints. You don't have to give the capabilities. You just say the endpoints. Like here, it's a WMS. You can give some tags. And then you get into the editor. And there you can configure basically

which tests are to be fired on your endpoints. And the terminology we use is probes. So you have a list of probes. And each probe is basically a request, mostly, and some tests on the result of the response.

So we can zoom in. For instance, by default, you always get a get capabilities probe. So it will fire get capabilities on your endpoints. But you can also, this is a WMS, have a probe for, let's say, get map. And then you can edit the probe with specific parameters.

And some in gray are already fixed because the services should always be WMS. So this configures a get capabilities request. Basically, you can choose a version. And then the second thing of a probe are the probe checks.

And those are the checks which are actually performed on the response. So for instance, you want to have invalid XML, at least, from the capabilities, or that doesn't contain an OWS exception, or that contains at least a title.

So in that sense, and if you add, for instance, a get map, it has some intelligence to figure out what layers there are. So you can select a layer and configure, basically, a get map request on all, in this case, 70 layers.

And once you have configured, the tests are run currently every hour. But this week on the Code Sprint, we're working on a test frequency per endpoint. And when something fails, you can configure an email,

and recently also a webhook to get notified. So for instance, I use a separate email box. This is one of the governmental WMSs. So they regularly fail and get back. And this is not even due to the WMS itself, by the way.

It's a network which is maintained by a governmental IT department. So things can fail for various reasons. At least, we get notified. So this basically was a walkthrough.

So how does it work? Basically, it has three parts, GUL check. There is what you just saw, the dashboard, the web app, where you configure your services and your health checks, basically. And then there's a runner, which is now running via a Chrome job, which

performs the actual checks. And it says plugins, because all these checks or probes that you just saw are actually plugins. And that's a nice thing. You can extend it as well. And these two parts are glued via a database.

And we'll get to that. So this is basically the, you could say, the architecture. The web app is what we just saw, the dashboard. And the runner runs in the background, runs the checks. And the database stores the results and all the configuration and everything. And it's a Python project.

And for the dashboard, we use Flask. It's a standard whiskey setup. So you can run it in NGINX or GUnicorn. But the preferred way, and I've seen it also on the conference here, is Docker. So if you're not yet into Docker, get into Docker.

It really works well. And the runner is now via Chrome. And it runs what is called the probes and checks. Those are plugins. It has results reporting. And what I said, notification. You can have notifications for the whole server. Or you can have a notification per endpoint.

So specific email, certain endpoints are maybe maintained by others. And they may have to get those emails. I talked about probes and checks, which is really the way that GeoHealthCheck performs

the health checks, the quality of service checks. So they're actually all plugins. And in the standard package, there's quite a few already. And you can write custom plugins. And when you have a plugin, you also

configure all its parameters. And that will be visible in the UI. I think there's an image later on in the database. In short, it runs via SQLAlchemy. And you can use preferably Postgres, PostGIS, or SQLAlchemy.

Oh, sorry. SQLite. Any database, basically, is supported via SQLAlchemy. There's tags, so you can group resources, because you can get lots of them. Installation is basically a standard Python setup.

But it is easiest using Docker. We have versions Docker images on Docker Hub. And we have Docker Compose. Docker Compose basically is a tool to combine several Docker images, one Docker image for your database. For instance, one Docker image for the geohealth tag.

And it glues it all together, and you run one thing. You could be up and running in minutes if you have a Docker environment. So various settings. I won't go in all the details. But you can imagine you have several settings in a file.

And for instance, if you want notifications and to whom it has to go, of which you have to configure also your email clients if you need email notification. Five minutes. A little bit about architecture.

We have a reasonably simple data model. Central is the resource. That's the end point. And it's related to a user. And we also keep all the history, which are the runs. So the model is really each resource has a URL.

And that's an end point. And the probes, they fire requests on that URL. You can have multiple probes. And a probe can have multiple checks. It's basically the checklist. And finally, it will create a report, which is now a JSON file. So you can format that in any other.

Talked about the plug-in model, but seen the time. But I would like to show some code. Because, yeah, plug-ins always sounds very difficult. But it's actually very simple. So this is the simplest plug-in, the probe.

This is an HTTP request. And it's basically a template. So you have to provide some parameters. And for instance, its name and what type of resource. So it could be a WMS resource, whatever. And which are the available checks.

And that's about it. And for instance, if you want to implement a check, this is a check which implements if the response that comes back from HTTP is not errored, which means it's not in the 400 or 500 range. And it's just a few lines here.

And then, of course, it can get more elaborate with capabilities. But it's still, there's no code here. It's just definition. Because it already knows it will do an HTTP request. It only needs to know what the template URL is. And basically, that's what you define here.

And it can be extended. And finally, it's also what you would see in the UI. So you configure parameters, let's say, a version. And you can select a version. It's now a dropdown.

Something about the project, it's on GitHub. And like I said, started Tom Kralidis somewhere in an airplane in 2014. It's on a GeoPython organization. Any people programming Python here?

I see some hands. Because there's many other interesting projects there. And on the Code Sprint, various authors or developers from these projects are actually there. So you can meet them in person and maybe cooperate. Because we have a demo running, demo.geohaltcheck.org.

And some main things are under development. There's an issue tracker. Because this presentation is online via geohaltcheck.org.

And we're planning, for instance, a REST API architecture. So you can also configure the checks from a distance and integration with existing monitoring tools. And please, we can always need your help coding, testing,

documentation. And luckily, we are receiving more and more help and PRs. But yeah, you're very welcome. Thank you.

OK, thank you. So are there any questions? I actually have two questions. First question is, why does this slide set 42 pages?

42 pages? I just checked. It's online. No, 42 is the answer to all questions and this kind of stuff. But apparently, you don't know. So I'm sorry. The other question is, have you been considering to get the error messages actually into HTTP with the OGC?

So is there thoughts at the OGC that you don't return a 200 if it actually fails? Because I know that there has been some discussion. Because it would make sense to say that the server answers, yeah, it's not a 200. I'm not sending you anything. But instead, it doesn't happen. So do you know whether there's going to be an alignment of

the error to the HTTP standard? Yeah. So the question is, is OGC improving or changing their standards to really support error messaging in HTTP?

I think they are. There's the most recent developments. For instance, WFS3. Is anyone following that? Well, the general movement is to go more to REST API type of standards. So you use HTTP, not just HTTP verbs, but also the

error reporting. And for instance, sensor things API is also more RESTful. But I wouldn't know about the existing standards if they are being modified. I know of some of the UMS servers, and I remember in the past, they actually would send HTTP error

codes, but most wouldn't. And at least we forced, for instance, to send an exception report. Because the most difficult thing would be to analyze an error which is written in the image that would be sort of artificial intelligence.

Wow. Maybe a nice student project or, yeah, sort of thing. I wanted to add an example which you didn't name in the beginning, which is a necessity all of Europe has, namely Inspire. Because the Inspire guidelines, they demand a certain uptime

and performance from the services. And we actually tried the GeoHealthChecker to monitor this about a year ago, but couldn't do so because it was a limitation of a ground laying library that WMS 1.3 standards were not supported.

I checked again, now it's supported. And I think it would be an idea for a probe to actually add Inspire validation of the response XML. Yeah, very good question. The actual question came up also on Phosphor GAU from someone from the JRC.

And of course, yeah, in Inspire you have a very specific quality of service requirement regarding WMS. And that's hardly to do with one of the standard probes, like a response have to come back within five seconds. And you have to have so many requests handled. And it would be a matter of developing a standard probe.

Yeah, custom probe, I mean, custom probe. But we're happy to cooperate, and yeah, good question.

A question, this looks really useful for system admins and so on to monitor services. But as a developer, I'm also seeing some potential to use this during development of services and so on.

So do you think it would be very difficult to extend your health check to use, for instance, for load testing, performance testing? Because you have the probes, you just need to run them. Yeah, of course. And do some reports, I guess. Yeah, you can write a custom probe.

I mean, there are two types of probes. I probably forgot to say. There's one that's a templated one, that you have one request and you have your checks. But you may have a look into it. There's, for instance, a WMS crawler. And then you can do as many requests as you have. And you can have checks that maybe say, well,

those responses have to be within that many seconds. And otherwise, it's an error. So you can write custom probes that would do that kind of scenarios.

Would require some Python coding. And use of OWS lib, I should say, which is very handy to talk to this client library for remote services. Another question. A lot of information.

I know I personally will definitely check this. And so we'll try to use it in my projects.

I can't promise that I will ever contribute to that. But we will see. So thank you for your talk. And thanks to the audience. I think it's time to get some coffee and refresh a little bit for the rest of the show.

Thanks. Thank you.