GeoHealthCheck: A Quality of Service Monitor for Geospatial Web Services
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Alternativer Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 295 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/43344 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
00:00
Weg <Topologie>MultiplikationsoperatorWeb Servicesp-BlockVorlesung/Konferenz
00:21
Web ServicesInhalt <Mathematik>TabelleArchitektur <Informatik>Web ServicesMehrwertnetzTesselationPlug inKonfiguration <Informatik>Befehl <Informatik>Arithmetischer AusdruckInstallation <Informatik>Rechter WinkelMAPProjektive EbeneComputeranimationJSON
01:27
VersionsverwaltungDemo <Programm>Physikalisches SystemFehlermeldungMessage-PassingSpezialrechnerExogene VariableHydrostatikUmwandlungsenthalpieWeb ServicesInternetworkingGenerizitätAussage <Mathematik>Motion CapturingKontextbezogenes SystemSelbst organisierendes SystemProzess <Informatik>Overlay-NetzAusdruck <Logik>Demo <Programm>PunktFront-End <Software>DienstgüteAusnahmebehandlungBildgebendes VerfahrenWeb ServicesProjektive EbeneCoxeter-GruppeCodeMereologieKontextbezogenes Systemsinc-FunktionCASE <Informatik>ViewerDatenbankElektronische PublikationMetadatenMultigraphInstantiierungTesselationPhysikalisches SystemElementargeometrieZahlenbereichRegulärer GraphServerMultiplikationsoperatorVerkehrsinformationSuite <Programmpaket>FehlermeldungSystemaufrufRobotikWort <Informatik>MomentenproblemRechenwerkDemoszene <Programmierung>HochdruckPhysikalischer EffektYouTubeSchnittmengeURLFlächentheorieMailing-ListeMAPSelbst organisierendes SystemComputeranimation
06:23
FrequenzWeb ServicesLoginServerAusdruck <Logik>KardinalzahlFluidURLNormalvektorKorrelationMenütechnikInklusion <Mathematik>Demo <Programm>Architektur <Informatik>SoftwaretestWeb ServicesMultiplikationsoperatorResultanteE-MailKonfigurationsraumElementargeometrieMailing-ListeInformationMAPDefaultQuaderSpeicherabzugPunktTypentheorieURLPasswortDifferenteGraphDatenbankZahlenbereichElektronische PublikationAusnahmebehandlungBitCASE <Informatik>BenutzerbeteiligungVerschlingungSchnittmengeKomponente <Software>Web-SeiteStatistikIndexberechnungWort <Informatik>Mapping <Computergraphik>SchlüsselverwaltungMittelwertMySpaceTermVisualisierungComputeranimation
10:53
MereologieWeb-ApplikationDatenbankStandardabweichungKonfigurationsraumSchedulingVerkehrsinformationDatenmodellAbstraktionsebeneURLExplosion <Stochastik>ChecklisteInhalt <Mathematik>Exogene VariableBoolesche AlgebraKonfiguration <Informatik>TaskKundendatenbankMigration <Informatik>Plug inErneuerungstheorieZeitbereichWeb ServicesArchitektur <Informatik>Desintegration <Mathematik>HilfesystemSoftwaretestTranslation <Mathematik>Pay-TVPunktwolkeQuellcodeDemo <Programm>Chatten <Kommunikation>Coxeter-GruppeVerschlingungElementargeometrieComputerarchitekturMereologieMAPBitGefangenendilemmaTouchscreenRegulärer GraphMigration <Informatik>Spannweite <Stochastik>Front-End <Software>InstantiierungTermPlug inResultanteTypentheorieAutorisierungSchnittmengeEinfache GenauigkeitTaskKartesische KoordinatenSoftwareentwicklerDienstgüteWeb ServicesDemo <Programm>Prozess <Informatik>CASE <Informatik>InformationBildgebendes VerfahrenKundendatenbankRPCVerschlingungAuthentifikationStandardabweichungMultiplikationsoperatorDomain <Netzwerk>CodeSoftwaretestDatenbankQuick-SortEndliche ModelltheorieTestbedSchedulingWeb SiteDebuggingp-BlockExogene VariableBenutzerbeteiligungCloud ComputingVerkehrsinformationProjektive EbeneSpeicherabzugFehlererkennungWeb-ApplikationKonfigurationsraumCoxeter-GruppePunktPay-TVAdditionHilfesystemAnalysisGruppe <Mathematik>DatenmodellFigurierte ZahlE-MailRobotikGemeinsamer SpeicherCOMQuadratzahlZusammengesetzte VerteilungTeilbarkeitGesetz <Physik>AbstraktionsebeneVolumenvisualisierungMinimumVorlesung/KonferenzComputeranimation
19:47
Plug inMultiplikationsoperatorRechter WinkelNeuroinformatikVorlesung/KonferenzJSON
20:10
DatensatzInhalt <Mathematik>CodeWeb ServicesKomplex <Algebra>GeradeProtokoll <Datenverarbeitungssystem>MAPPlug inBitInteraktives FernsehenFunktionalRuhmasseElement <Gruppentheorie>Ordnung <Mathematik>MultiplikationsoperatorSoftwaretestGrenzschichtablösungOffene MengeInformationTVD-VerfahrenWeb SiteRandomisierungQuick-SortNummernsystemBildschirmmaskeFlächeninhaltHilfesystemEndliche ModelltheorieVierzigParametersystemSpieltheorieProgrammiergerätInstantiierungDateiformatZoomGüte der AnpassungDatenstrukturKonfigurationsraumTesselationProgrammbibliothekValiditätLastParserHill-DifferentialgleichungElementargeometrieKonforme AbbildungJSONXMLUMLBesprechung/Interview
Transkript: English(automatisch erzeugt)
00:09
Okay, we can get going so thanks again for joining us for this last session of this block For this talk we're going to talk about keeping track of the uptime of your
00:22
Geospatial services with Tom Kralidis and Jus van den Broeke So I will let you go Okay, thanks Daniel So basically Jus and I will walk through
00:40
What the project is all about the problem statement will give you an overview of the tool and How it's put together how easy it is to install options for extensibility and plugins as well as what the future holds So I will first take you through some what we call OTC OWS
01:04
monitoring challenges So has anyone of you heard this expression? I see pink tiles Right, yeah, okay, so you all know what this means
01:22
We expect something like like a map being displayed, but instead This is what we see Why? it's a surfer suddenly sending pink tiles today or Probably you know the answer it is
01:40
somehow the surfer is failing and it's sending some Well-formed exception report for instance and in that case this was open layers, and it doesn't really know You can even configure if it must be pink or any other color, so
02:01
I'm trying to make a point here. I'm presenting actually some some challenges even problems we have running OWS services because in Another so this this is a WMS request But you can also configure that instead of the well-formed exception report you can have an in image error
02:22
So why am I showing this? Oh, yeah Images could be blank Like you have the backend database which is filled every night, and it shows maybe some overlay But the ETL process fails so you have an empty database, but still valid images although. They're white
02:44
transparent The point I'm trying to make is you want to be aware or notified of Such failures and in many cases. There are what you could call HTTP monitors like
03:00
Maybe you heard of pingdom of time robots or or any other We've seen this in also the other presentation that HTTP is basically used as a channel so with many of these OWS services you will get a 200 which means an okay code
03:21
So a regular uptime monitor will never detect these situations. I just showed will never detect pink tiles or exception reports It Goes further the OWS serves usually have something like get capabilities and It may be okay and some uptime serves. They can they may check for get capabilities. Maybe even for some keywords in there
03:47
But it doesn't guarantee that any of the more detailed service will actually work Like you have WMS with 500 layers and layer number 397 is failing. How would you detect it?
04:01
Because I've seen also case where the capabilities file is actually a static file or generated fires and mother metadata system so It goes further time-based OTC services like SOS Sensor things API
04:21
You may have this is a viewer for instance you may have gaps in your data because Maybe a sensor is failing or some something in the entire pipeline And you wouldn't be Noticed unless you look at the graphs all day
04:41
But you would like to be notified when when this is happening so Like I said you can you can run public uptime services or or internally but they usually only check on HTTP So
05:00
What we actually need at least what I needed and that was one of the reasons I joined this project is OWS aware service monitoring and More Detailed or said quality of service checking
05:21
and since things may Fail and then come on back up You also need history capture, and I think this is where Tom will take over and lead us through geo health check
05:40
Thanks just so an overview of the project I was delayed on my way to Phosphor g 2014 at least the code sprint and it started somewhere on that on that flight path, so Good things happen in the air that day at least for me, so it's part of the geo Python community of projects
06:03
So if you go to geo Python github.io, there's a slew of projects many of which are being talked about today It's part of that suite of projects So a quick tour You can go on demo dot health check org at any time which provides you a live demonstration
06:22
So here we see a dashboard of Basically you know a number of services you can you can see You know very high level how many services are operational how many are broken and so on you can see a little map of the locations of the actual services as
06:40
Well you you have a list of services you can filter the list with the search box You can get a dump of the list at any time in JSON or CSV It shows some very high level information around the last time the service was checked What was what was the result some some average some basic statistics and an overall reliability?
07:01
Reliability indicator as you can see on the left-hand side. We have various resource types so different types of OGC services and and and other things as well as keywords and and settings and so on So here we filter by by web map service so you can look at only the WMS is in your in your deployment of
07:24
GHC and once you click into a given resource or In this case of WMS it gives you a deep it starts to drill in and tells you a little bit about the service All the all the high-level information, then you can see a graph below which shows
07:41
You know red is obviously when it was when when the checks failed and green is when they passed and as well how long it took to do to do the Given checks if you wanted to add service you can just basically log in you have to create a username and password And register after you do that, then you log in and you can
08:04
There's a add resource There's an add button up at the top of the UI and it gives you a bunch of different Things resource types you want to add so if you wanted to add a WMS for example You add the WMS into your add resource so very Straightforward kind of wizard to way to add something to GHC at this point
08:28
We have a concept called probes which are once you add a service into geo health check there's a there's a Various level of granularity of the kinds of checks you can do so there's a default check where it just doesn't get capabilities
08:43
almost like a smoke test if you will then there's a bunch of other checks which if you want you can You can choose to add them, and then there's extra extra checks being done So there's another example of for example some of the default values that you can set when
09:02
When interacting with this when doing the tests against the service You can also do it you can also configure the checks to Whether you want to test every single layer on that service, or you just want to Test certain layers against that service, so that's the the nice thing about what we have here in geo health check
09:21
Is that you can configure it to do very specific things against the service so if you know that A specific layer is powered by a database And you want to see how that layer is performing you can do a test against just that layer and making sure it actually does What it's supposed to do so more examples here of pre configuring how you wanted to test the service and then?
09:50
When the services fail we get an email Configuration there's an email configuration component in the in the in the geo health check so we get an email Which tells us look this service failed and here's why it failed and then you click the details link and that takes you to
10:07
A page like this. I think there's some debugging information here Yeah it tells you why why the service failed in this case we can see from the service exception that some file was missing and
10:23
Then when a service is fixed you get another email that tells you it's fixed So it's very much if you've ever used Travis that sends you emails failed fixed still failing and so on it Follows that same kind of spirit, so I'll just move it over the juice now
10:40
Yes, Thank You Tom and by the way also Additionally With email you may also configure web hooks and Quite open I will lead you a bit to the high level architecture so there are actually three parts in geo health check there's a
11:04
Dashboard which is a regular Python web app written in flask we just saw but in the back there's something we call the hell check runner which Well runs the hell checks actually and it's as we can see
11:22
Totally built with plugins for all these probes that we just saw and there's a database and Well it's a simple high-level architecture where they are Related in this way, so you configure everything to your Web app and the runner will take that information to do the hell checks
11:45
So the dashboard we just saw Flask web app you can do standard Python They ask ie whiskey it's also called or run it with nginx Apache 2j unicorn
12:01
standard Python So the hell check runner runs style checks, and it's driven by a scheduler because You can also configure how often you want to do these hell checks and some you may want to do every 10 minutes and
12:21
So this process this is basically a backend process or you can choose also to run both the Hell check runner and the dashboard in a single process And it takes care for reporting when it's up or down or you can even get a daily status report which is also sometimes handy and
12:43
We have different ways of notifying so there's email, but also you can configure web hooks to other monitoring remote monitoring applications So the hell check model is really Yeah, we built in some abstraction here, so what we call a resource is basically your
13:06
your endpoints identified by URL so that for instance WMS endpoints and then the probe is really the Thing that
13:21
Initiates the check it fires Requests on the endpoints But doesn't interpret them because that's another Aspect those are the checks so you can configure probes and for each probe you can configure one or more checks and For each resource you can configure one or more probes
13:46
So Let's see So the probe has end checks each check evaluates a single result so for instance if what comes back is a HTTP
14:00
Error code in a 400 or 500 range that is typically what a check could do or if WMS response is really an image of type image So and finally It establishes a run reports and
14:22
Each probe or check is is a plug-in so it's a plug-in because there's no ends really to what type of hell checks you would want to do so It's built of plugins from from from the bottom basically now and the database
14:42
Will basically has a data model, I think it's shown in here With all the entities that are involved. It's not very complex and because we also have users There are also users involved because users they can be added by configuration or you can configure that any user that like on our demo site can do can register themselves, but
15:08
there are cases where of course you wouldn't want that and I think Tom will Take us Through the installation
15:23
So it's a standard Python setup with setup and management tasks You can also use Docker which allows you to easily install as you can imagine There is a look at some of the core settings and basically a Python file, and you set each of those as you wish
15:49
I won't talk too much about this given the time we have left, but basically we mentioned the fact that there's plugins so you're allowed to create your own types of
16:00
checks and probes within those checks, so if you have Some sort of API which isn't even geospatial you can you can write a plug-in to You can write a plug-in to do the test that you want and do the probes that you want So it's fully extensible, and you can define those In your Python code and then a UI shows up and renders it also
16:22
It's fully flexible in terms of adding new types of service API's on into the ends of the health check roadmap It's always fun to think about the future I guess So geo health check is is looking at doing a rest API for the actual
16:45
Things that need to talk to the back end so that we can have a really thin front end right now It's a little bit of a mixture, so we're looking at refactoring that we're looking at Vue.js in terms of updating the the UI We're almost done the Python two to three migration is just like we've heard in some of the other Python project projects
17:05
and we're looking at integrating it with other more mainstream monitoring tools and In addition at the OGC level there's also something called the quality of service and experience domain working group So we're loosely involved at that looking at how they're working on particularly the the QoS
17:24
Things and I know that geo health check has been used in some of the test beds in In OGC so whatever work is coming out of the OGC It's important for I think it's important for us to look at that work and try to support it As much as we can and you can help so I know you've all forked the project now
17:44
There's there's lots of issues And there's a lot of enhancements a lot of work can be a lot of help can be with the documentation So we're certainly open to contributions Oops, I'll move it over to use at this point. So in case
18:07
Shameless block here You don't want to go through the well. It's not so much trouble of installing It's what maybe it's nice to have something which is really outside of your infrastructure
18:20
We I've also run a let's say you call it a cloud service a subscription service called geo cars dot-com and Then basically You get what you get is really a geo health check your own geo health check instance, and then you can configure
18:44
Whatever you your resources want you can also shield it. It's with Authentication so yeah, I Think we're about It and here are all the links to the to the project
19:01
So the main link is really geo health check dot-org this presentation is also online we can also because there's quite some details still on plug-in developments and For instance we had a great contribution from the geo node community I think it was geo solutions that had a
19:22
Probe for a geo node backends because you know the also provides Connects to lots of other BS's and that to do that all manually so that's the kind of contributions that that we really welcome and
19:44
Thank you If the I think all three authors are here Tom my analysis On this writer is also he was here in the last year is And there's more contributors of course for the for all the some of the plugins and so
20:04
Thank you Thank you very much Tom and juice. We have a few minutes. You're right on time So we have a few minutes left for a few questions. Let's start here Sorry it is possible to perform
20:21
some basic bulk loading because if you have hundreds of OGC services to check It would be useful to load them with maybe basic checks in order to start a basic performance test
20:43
So just checking the get capabilities for example That's a good question we did have some functionality in the back which allows you to bulk load But we can we can bring that out a little bit further to Like basically let you run a command line to you know load these
21:03
hundred services kind of thing and load them and let them be tested kind of thing so Feel free to open an issue because it just needs a little bit of work. It's not I Think there's actually an issue open for that now. I understand this is it's for bulk loading configuration of services. Yeah
21:23
So if you have a WM TS It's their functionality to test each zoom level good question w We have something for TMS But I don't think we have yet for W MTS also because it's a little bit complex
21:46
protocol, but that is typically a Contribution we would welcome So basically you have to do new plugin Yeah, but you could use the existing because there's there's for instance in probe which allows you to do any
22:06
HTTP request so that could be a a request to a WM TS, but But not yet in the sense that we have for TMS and for
22:20
WMS WFS even WFS 3 or I should say OTC features API So hi I want to know I'm not a programmer, but I need to make some Personalized probes for instance to know if the WMS has every parameter completed like the name the contact information
22:46
That sort of thing Is it really hard to do do I need to know Python to be able to do it all from the UI? I can do it
23:00
Yeah, yeah, the question is really more about Conformance it like sounds like yeah, and probably There's something called site tests from OTC which is more suited for that you don't have to program them yourself Tommy Tom knows more
23:22
You'd probably have to write some Python like what I'm hearing is you want to look for specific Elements and yes, they're there and what's in there and everything because we are thinking about implementing the hill Check to be able to know if the WMS has every element the national SDI requires and
23:41
We don't want to use many many somewhere or other Yes, we don't want to install several ways to to check it if we can do it all in the hell health check Yeah, so you can use I mean I would you can use a you'd have to write code
24:01
But having said that the code that geo health check uses underneath to interact with all those things is a tool called OWS lib so most of that functionality to read that Content model is there so all you have to do is write a few lines on the checks Uh-huh, and so you're it's almost you're halfway there, and it's what you ought to write Python
24:22
The parameters I need are the in the OGC Documentation or in the live you should be in the lip, okay, yeah Thank you. Maybe under suggestion is to use XML parser you can call and then do schema validation against the XSD
24:43
schema from OGC But that's good one. Yeah, if you do a health check on the WMS Do you then always get like the same? Tile or do you random move between?
25:03
Also a good question actually we have a Structured way of testing WMS like get map and but it will always do the same area, but there is also It's a little bit experimental. It works. It's we call it the
25:22
WMS drill down and that does this kind of random behavior Which sometime so yes, but that could be improved a bit this You can see it in plugins. It's called WMS drill down but it will take random areas and
25:41
even random formats, but it's Usually fails because WMS says well I support these hundred formats not and I try we tried one and of the the random one so yeah Probably yeah, it takes some random whatever that WMS said it supports. We take some random, but