Geodatenmanagement mit GRETL

Video in TIB AV-Portal: Geodatenmanagement mit GRETL

Formal Metadata

Title
Geodatenmanagement mit GRETL
Title of Series
Author
License
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2019
Language
German

Content Metadata

Subject Area
Abstract
Beim Amt für Geoinformation des Kantons Solothurn steht seit 2018 das Datenmanagement-Tool GRETL im Einsatz für den Datenimport und -export in die bzw. aus der PostGIS-Datenbank, aber auch für Datenumbauten von einem Datenmodell in ein anderes. GRETL ist ein Plugin für das *Gradle Build Tool*, wodurch die volle Power eines Build-Tools neu für Geodatenflüsse zur Verfügung steht.
Loading...
Manufacturing execution system
Server (computing) Spatial data infrastructure
PHP Server (computing) Structural load Error detection and correction Spatial data infrastructure
Plug-in (computing) Degree (graph theory) Building Software EXTRACT
Plug-in (computing) GRADE Code EXTRACT
Hospital information system EXTRACT
SQL EXTRACT SQL
Plug-in (computing) Error message Link (knot theory) Download Server (computing) Computer file Jenkins CI Task (computing) Row (database)
Histology Software repository
Maximum (disambiguation)
Inference
so welcome it comes namely at andreas schmidt weber geodaten management with gretel tell the name is very promising and what curious what that is behind it hides many thanks yes hello to say my name is andreas schmid I'll tell you about straight management with gretel nice that I still Some have come here too this late hour short for me me am gematik engineer and work at
office for geoinformation of the canton Solothurn in Switzerland I have you as the marked on the map so latour We are located in the canton of Solothurn to evict the cantonal geodata infrastructure and Although we have been doing this for about 2001 open source software a that started with web server and Post is someday is cookies We finally got there Google-server switched and more recently is also a new web access client application online.de on cookies web client based and because we because some Things are already very old we are all around and there too
the third is the idea for the kilo Added data management I have it more of a quasi initial situation we already have around 15
years the previous one geodata infrastructure in operation and on aspect of it whenever the data import and Data export scripts and the goods were after this was easy for a long time no longer overview was on the one hand were they are, they are still in different languages written there was there was surrounds script and php script mostly in ancient versions At some point, both scripts are added come a very important import
written directly in java and there are probably still some some spoke more to me have since not not found then The imports also use the power different tools if you call any external commands for example, whether yoga or sap sql um data from a shepherd dog post in the ordered to import database many are the logic too somewhere in database functions on the Postal bank hidden or even then somewhere in complicated views be materialized in some way what the overview unfortunately not beneficial and then the scripts also have different quality standards on just re walking or error handling and then there are also concerning surveillance These forces are very exemplary are programmed and others the for example, exactly in the critical moment has nothing to lure and to Last but not least are all these scripts and jobs on different runs different environments mostly in desktops but the first time on the a server has been installed Sometime the newer ones on a new new server installed and they are also about different user accounts spread because you cry a certain just over the reason have one wanted to run certain users So that's right with the permissions but as already said about any one you have to find a specific job first looking very long
to's are here the well-known children's song and that fits which looks very good on our situation too
we fall in love somewhere in one certain sense in the forest and it was like that dark and bitterly cold we have someday we just can not do it anymore rightly found and we noticed There must be a new solution here first obvious variant would be We have been more so far but better So everything was new have to write as good as possible quality But we had the feeling there it must something a little better something other and have us looking for and then slowly it has become this other solution and I signed off have to say this is really cool Solution and me a same third me Come on why that's what we're called have realized that such a geo data flow general data flow or even a data import and export in all usually a chain of individual is what's up there in the above graphic so symbolically represented and that there is an analogy to build here software also gives you if you have one software will be published first compiled then run tests then would be the packages will be uploaded somewhere and provided and so on and our credo was that we possible just want to build on existing do not want to reinvent the wheel and and then are on the cradle build tool encountered that I url has stated is just a tool originally to make software and the big advantage we just saw is that just that that this chaining logic takes over when a step fails the whole beats job is missing and that is what we want so not half things for example, a classic come out case in the old scripts it will First, the content of a table completely deleted because he has new data but if the arrow that is there imported he should or just not there then stays connection to teacher's table back and Such things just for lack of complete monitoring of the import Most of the time we have scripts noted if a customer has complained because because he found a teaching table has to save now We have a quarter as a plug in for bradl implemented bradl can be expanded with plugins I have listed the watches where you are that a brief information about this blocks as you just then used and so I come too Names Brettl is just an etl tool stands for extracts transform load the
certainly know some just a tool that can extract data from somewhere she can rebuild and back into one other data into a destination data source can load and it just builds on grade Run and that's why we're on the name Gretel came across
The code of von Gürtel is found on the tap and the software is under the apache license the one open source license is now on the juvenile features of battle though
The battle offers so-called apps on First of all for import and export of data what is virtually the extract and loud Part of von rtl covering is there first once a stone the the dbu de base tab With this one can get data from one Copy database to another we use that very much again often because we have so far actually with almost only one is one huge database the work new Let's break that up and run one collection database and one publication database and there are Accordingly, many jobs just the registration data from the registration database in the publication database then copy another one very much important very important part of gretel are many interviews steps matthias kuhn has in his contribution you always talked about interest I also see that I want to be short almost take the star entry and it is a description and transfair with transfer mechanism for geodata that just one Swiss room is the since 2003 exists in an earlier version even since before and this one interview tips the benefits ezb go that matthias customers too that too , especially in his Post occurred with elite can that is with me to say at the last place quasi a scheme import according to a left model that is the import the climate is Apple can also do data according to the import model or model together from which export from a schema And then it even offers in Europe
Ace in the lip update to only part import records and then there is the elevator of the serves interesting to us come to validate whether you a particular left model life correspond then there is almost the analogue too for the csv format csv export csv import and cs rivalry also can be cs we are against an internal model validate that is still special but very elegant then that comes good old shape filed that we too there we have taken it with us not even used until now I hope that will stay that way and that is in the next but one part outlook theo package is supposed to be the ball as well to come to and there i see you my boss had something new recently raster export to just rest or had which are available in a police database in an on-site picture too
export then the other part battlestar so that is a good day for the data around Some transform part of from rtl is the so called sql execute which can within a database, a data Make remodeling by simply giving it a hand
Any sql can give that then he on the database carries out there you can then maybe not so many you can do obvious things actually a whole database to use content somehow fill with content want to start working with these Tips is just an etl Tool I will be a database centric because it on the functionality of one Database based on Husky's database and also because the language of our data conversion from one model to another easy to achieve model sql is that has the big advantage that you can not another language or one certain software must then learn if you can as sql you can daten make tags with gratton and to emphasize and very very useful I find the separation through this individual tips of data import and data conversion we have in our old scribbled that partially mixed and that actually has almost only disadvantage and then it's nice modular because that it works with the individual steps So it can be very problem that expand by single steps Now what does a third job look like? have since virtually the helau example gretel
compiled at the beginning there are one of which has some dependencies just straight intel is to find unity which it needs and there in the plugins area one indicates that one just that then still wants to use flash plugin a 1 2 in odds and then comes actually the 1st so the real task I have einhell called rattle you have to specify give him a name which guy he is it is a db2 db2 then you give it to the internet connect the connection to the source and social table to target database with emotional password and can him then one to several transfers to hand over transparent means a sql query which must be present in one case this will be data from the source data Bank brought and then you give him to the aim the goal theme goal table where should this data and with when there also two behind depends then defended goal here is learning first deleted before the dates then be imported down two examples as just this look sks They can be quite simple in I even even have the second one here's a little data around but a minimal data built around bahrain is shown by the column 2 1 nor with multiplied by any value and then do the thing you call the command line bradl on you have to just has installed before and behind as an argument the name of the steps should execute the bettel gradl picks up then because we on the yes at the beginning the plug-in and all needed dependencies then on to the features we but not that we did not program all of it fill of features just Gradl brings along there is for example a copy paste to copy from files and directories or even to tap and be delighted with files then it offers of this bradl control of dependencies of tasks with each other I have an example of the concepta here the one of hobbys depends on any previous step down load data from then bradl also takes over It can be done with loops Integrate over single part record and because he's using plugins expandable we can we have again much much more task to disposal eg we use the download task from a download plugin straight a link to a larger one there I do not want to detail it enter I leave I that easy to foils with recorded around the case if that
someone wants to look at a job himself have some steps in a row performs with these dependencies then we need another component for the control of all jobs that there were yes shops so far and we need something that just the ertl jobs triggers and how we, too Here again our credo was we want on existing excitement but nothing let program that there any jobs started and we started have found there jenkins because jenkins 90 is called automation server jenkins you can next to indicate to a particular a certain time should this job to be performed jenkins then checks out the third job From a repository we have all jobs in a gate and and lead him then out of benefits of jenkins is First of all, that the champions the loks and the course itself that keeps everything that way transparent Jenkins offer also offers user registration rights management for for example always something in case something something silent and we have here all jobs in one place all users have a login for jenkins can look there what with the jobs is going on
short some pictures like that looks in jenkins that's the overview of all jobs here the view of an individual jobs and left-wing just outside the course the last versions listed you get at the red dot sees that because something went wrong who clicks on it and the consoles issue shows you just get the The lok edition and jenkins also provide
the so-called blue ocean surface So doing it with us is something the simpler surface still the first still looks good
somehow feel and the gretel jobs in I have the clock in a pot repository then stated here again benefits this is visible to all that is so publicly available there differentiate of course the Traceability of changes thanks to versioning and here too we already have all the job definitions a place that looks his job
the daughter is the picture gradl that is the central component and then there are still just scribbled this picture needed there I am yes it is the
second last, we operate all of wolfenbüttel positions double container and the whole thing is going on then surrounded in an open open shift jenkins as a double container and gretel From dr container we have until now good experience with it and then my conclusion so far this really works as robust and reliable is very clear flexible, easy and expandable then with the step concept we are absolutely satisfied and will be there to develop even more
and we are happy if that too maybe from other users is used then thanks for the
so many thanks for the attention Exciting lecture, there are questions
thank you looks very interesting if I have the right head allowed jenkins is also an event based to do things makes everything about regular
orders or do you have logic in it if data comes in then generate something else passes we have not been that yet Such a good part are regular regular state just about a cure and off and the other part are jobs you only manually can run as one jenkins must log in and there click a button to start emanuel but the other was was quite possible that jenkins supports just yet thank you a second question maybe i have for post the other side is already used or so far only with you because I have the feeling that is something really many people could be interesting yes is busy
only with us but I hope that that changes after this presentation Yes yes my question who sets only for internal data streams 1 or is it also for an export portal So for a colleague of the exports manufactures via a web portal callable basically you can jenkins absolutely operate in a web and from the streams of data here yes it is up to us now only internally but we become the example from an external fdp soldier but we have already got from a year died ftp server data if and just uploads on fdp but be do we, I also think the databases could somewhere alone the Databases could be external somewhere if that allows to access they said before that one Also rasta tatmesser from a post is you can show and everything card can vice versa, for example, too a motive in the database with it Nice start, there are efforts facility or I do not know anything about aspiration but I assume that that relatively easy to implement ok then i have another question and Although such discrete liz came up with Spatial data in any case but it is also not possible Spatial data can be imported I understood it that way or right yes absolutely so that first heard we will not often spatial data or data the maximum an xy coordinate but deliver it is absolutely not meant for accessible data usable ok then if no further questions we ask Then the session is over and at 7 o'clock it continues with the Thank you
Loading...
Feedback

Timings

   79 ms - page object

Version

AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)
hidden