Logo TIB AV-Portal Logo TIB AV-Portal

Towards pycsw 2.0 - project report

Video in TIB AV-Portal: Towards pycsw 2.0 - project report

Formal Metadata

Towards pycsw 2.0 - project report
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
pycsw is an OGC CSW server implementation written in Python and is an official OSGeo Project. pycsw implements clause 10 (HTTP protocol binding - Catalogue Services for the Web, CSW) of the OpenGIS Catalogue Service Implementation Specification, version 2.0.2 and the upcoming version 3.0. pycsw allows for the publishing and discovery of geospatial metadata, providing a standards-based metadata and catalogue component of spatial data infrastructures. The project is certified OGC Compliant, and is an OGC Reference Implementation. The project currently powers numerous high profile activities such as US data.gov/geoplatform.gov, Integrated Ocean Observing System (IOOS), US National Geothermal Data System (NGDS), National Oceanic and Atmospheric Administration (NOAA), USGS Coastal and Marine Geoscience Portal, US Department of State, US Department of Interior, Greek National Open Data portal (geodata.gov.gr) and the WMO World Ozone and Ultraviolet Radiation Data Centre (WOUDC). This presentation starts with a status report of the pycsw project, followed by an open question/answer session to give a chance to users to interact with members of the pycsw project team. This presentation will cover the main features and enhancements made to pycsw last year (1.10 series), the upcoming 2.0 release as well as the future direction of the project.
Meeting/Interview sources mid sort metadata
report Computer animation attitudes report
font mathematics Computer animation Meeting/Interview bits font
Meta email suite MOF PROM unit formating clients open subsets mechanisms case repository record systems proper Development The list branch Transactional Syndication processes model point Open Source response component Integrationen catalog architecture specific Dublin Core profiles terms configuration services implementation protocols platforms web default standards focus Graph suite server cores clients applications limitations system call Software functions string CAP versions utilizes des classes libraries standards graded time ones part Transactional Volltexten Location geometric memory Operationen Query extent presentation Bugs email formating Temporal files Open Source open subsets profiles Types evolution interfaces configuration website testing since record digital filters implementation server mud encoding lines metadata versions operations environment testing default Operational Research addition noise CAP response regional forces Databases catalog predicate CAN-bus Computer animation platforms services extent
presentation Graph map time combination analog open subsets functions structured data geometric mechanisms Query box processes extent libraries parsing mapping Temporal Development argument open subsets mechanisms category Representation reduce spacetime point link files Integrationen maximal Continuation catalog The list Continuous Integration Content elements versions specific profiles Payload Graph Magneto-optical drive catalog Transformers Location words kernel Computer animation functions versions box services libraries extent
components response images Temporal open subsets clients functions metadata architecture geometric goods mechanisms profiles Software cores testing extent Abstract images Temporal clients bits indicators Types Computer animation functions SoC architecture sort Results
script sources link Free Computer animation indicators rules
presentation install code time Databases open subsets part variance web data management geometric Types information systems God sources Development Open Source statistics registry open subsets data management dual system website testing systems since server mapping content management systems Open Source lines Integrationen directories Hype Emulation versions environment testing services implementation Cats web default default standards server forces states experts catalog CAN-bus embedded systems Computer animation versions platforms life libraries extent Elastic
registry standards groups presentation server link time Guide The list clients heads metadata vision Meeting/Interview workgroup implementation determinants CAMS sources email The list smart Databases Transactional lines sets wikis Computer animation Software Blog organization WPAN editors Right log files Geodateninfrastrukturen Free localization libraries
Computer animation
but well I phase and where come to
this session I hope you enjoyed your coffee and on now well prepared for this session on metadata the and from our 1st speaker is so I'm just sort source and you will provide an overview of recent enhancements and made to apply
CSW please where command lots of people so that
will come over 1 this is a project report for fighters W and actually the
titles should be a bit in the change because prices W 2 is actually released now now but anyway we can this is the overview of the outline of the with today's talk we're going to
to go through some introductory in details so we're going to see the features what's new in the new
version of fighters W and then we're going to focus on downstream projects and future developments so 1st introduction so by the of w is an odyssey server implementation it's written in Python it's open source under the MIT license it runs on all platforms that support Python and a party so I also always has been and always your project since last year so we officially graduated so our goal from the beginning was to fully implement the Open OpenGIS catalog service specification and a lot of all the way we implemented some more stuff like OpenSearch recently and the main goal of spices W is to allow everyone to publish their geospatial data so that any other application can discover them through this year's API the project has been certified by OGC since the beginning and at some point 2 years ago we became a reference implementation of C is w version 2 . 0 . 2 this year actually 1 month ago the Odyssey released the W 3 justification and that the same day we really spices w tool which implements his W 3 specifications so this is a very short overview of the project history started to in 2010 by Tom Crowley this quiz and grew cattle in metadata I joined the team in 2011 and it was more widely announced and and then we had the release version 1 in 2011 than things to off from there we were included in the noise you alive we were then certified with to then we minus to get spices w deployed data data of golf the official US data portal and then we today we have the release of version 2 and the reference limitation from Odyssey for forces W 3 so our main focus was from the beginning to provide a very lightweight library that can actually plug into existing databases of metadata and to be able to be as lightweight as possible and I mean in in terms of configuration and memory usage and how fast it goes and etc it's very extensible you cannot be data formats was provided by the mechanism so you can write your own profile of fires or whatever as you you might have and but from day 1 we said that we're always going to be looking at the site test for all to see so are test happens through sites testing and are on the unit tests as we'll we're run every day so some of the features are not going to mention all of them because it's a very long list W 2 . 0 . 2 and W 3 is supported we are and how we can harvest that WMS WFS WCS and other CSW catalogs we implement the i zone with that the metadata application profiles the entities profile and we 2 years ago we implemented actually were we were the 1st implementation of OGC OpenSearch extensions and so we support transactional could capabilities and the figurative candles distributed searching and many of many more and so well I'll just go so that we are very focused on their own standards so In order to be able to be compliant with his w we also had to be introduced to implement some other standards so this is a list of standards that we support along the way some other standards so came up in there in the in the process like and nite and they said the firm and the as you an open search so these are the SEW operations that are mentioned in the W 3 and specification actually only existed in this year's w that 2 under without the and harvests 1 which is a new addition now so the server will ask for pitiable the client will last for kids capabilities from the server and then after it knows what what type of system that will be after he discovers the API he didn't actually ask for records or ask from to harvest the catalogs and stuff like that and you can also do transactions so so it's so what's new in prices W 2 mainly we were waiting for all this seem to release the the sea is W 3 specification in order for us to really spices W 2 and this happens because of this he take some time to review the documents and after the actual implementation happening is that it takes some time to to to release the the specifications so we were waiting to release our software to and so will pursue W 3 L better aligns with existing and new world notices standards like it supports now OWS common version tool and filtered version 2 it is highly based on all is made based very extensively based on open search an API now nowadays so as his W 3 and the open searches are coming together in many aspects and also although we has support for querying why it has to be a a temporal extents and distributed searching is much much easier for for their catalogs some more detail some other thing conference also includes a simpler achieve EPA API the role make metadata responses for get records by a the uh requests and also this year was missing for many many before long so long dialect it's DP status calls were not now properly used in the expected ensues W 3 and right so it's all so Odyssey are provided in the stomach of compliance test suite that was developed and actually follow with spices w 2 so while we were the implementing since W 3 that there the tests were also progressing along and we could help each other at some point and also there's another reference implementation so after a new version of tests was available we were able to just test our current status against that site testing and and figure out if if something is broken in our implementation or not so it it there this process can happen for of the year and set at some point this is the testing suite was finalized so we were actually the 1st ones who made in the mailing means that we fully past the compliance tests and well at the time of release and all this he now has to reference implementations will ready so what do some more the this involves the use W 3 now the default version of mind prices w when you ask the catalog it will default to see W 3 because this is what the specifically the what the specification we're still back work about compatible with the 2 0 0 2 version but you have to specify it in order to get an aversion to response so all this this new behavior of but actually brokering internally API and actually this was only 1 component of broken this these parts WCI and this is why we actually needed to release a new major version because it ain't broke the API and so on this is how it works now you just send me your
belly your horse name and last w to the server they're the new specification will not return narrative or was it something that everybody wanted that happens in instead it will send you back at capabilities documents and always is something that we all weighted and if
you don't specify version also it won't give you narrow had CW used to be very very picky about what to include in the in the URL in when you send a request so now if you don't specify the version you will get also 3 point 0 . 0 people with this document and then give space specify everything you get what you are requesting so this is something we expect so that also open service was a big thing for places W 2 the villages see also implemented profile for with Jewish in time extensions on top of the and w 3 c specification of open search and that this is a very light mechanism to query a catalog and using its STP gets so now we we support that actually we support that since version 1 . 10 which was sometime ago entities in the integrated in this W 3 also so basically OpenSearch profile gives you these 3 major categories of how to how to please the edge in a cuckold so you can just search through keywords or through bounding box or through time extent or in any combination of those so this can happen like in 1 1 greedy so here are some some examples for
example this is how you you can query the kernel for a bounding box it's it's much better than the previous version it was longer anyway so these are some some examples and this is a key word and bounding box request and we have links here for more more examples if you are interested to to test leans yourself the whole presentation is underprices w based so if you go to publications you'll find for for the 2016 there was another major things that was needed for for Python w was to support Python 3 because python is moving into Python 3 and it was needed to I had to be there so we implemented all we switch to to using future and 6 library in order or in order to be able to work with Python 2 in Python 3 at the same time and also we are not doing continuous integration through Travis on you have using 3 versions of of Python so we are pretty sure if something breaks or not when we are developing new new features but another nice starts to this release I think it is the better adjacent support but actually there's that the specification doesn't say anything about Jason but we used to be doing some mapping between what we what the XML elements where in XML file and uh we were exporting that 2 adjacent so that other developers would just use Jason would be able to work with it so now we know that we use in a very nice library called XML to data and then we managed to get the various compact output and better transformation of full of adjacent so this was before this will if you request prices doubled for Jason this is
what you got but then it is a bit awkward because repeats things like text and that it's it's and very XML like with Jason and now this is the same result which feels much more or much better for everyone who uses sensation so this is a this is a good thing edition foreign Jason people so are more features we are fly harvesting WMT we did some improvements in the XML output mechanisms we actually support this week's metadata profile because he would it it was contributed and also we have the better we have temporal extent support w mostly later harvesting which was not their previous releases so just sort of overview of that detector of how this works we have applied which implemented things like and indices and all the other stuff we have the CW core within our pull based and open search in history was different within the core court then we have the client school requests the metadata and this isn't a typical set up for the architecture was we are using as colostomy safely and Alex
MLW asleep and by-products of as dependencies over applies only any type of basic test a light as the meet supports and then we go we communicate with the external world through CGI or w z i and the party or whatever else you want to to
use the installation is very very easy to do actually we
have been included in beyond there are some discussions about all deceased my licensing which is not compatible to the to the uh the to the dead and rules but by CW has also reasonable and do all the there of course Python pockets indexed and then we're also and have and actually there's an easier way to in solid the if you use a script it will set up prices W 4 minutes
and if it takes 4 and a half minutes it's OK if you take standards not something is wrong but this this is something you can use for doing development and also by w has been always you
life since version 5 and 5 which is like 1 5 years ago so sorry so when the prices w being a library is very easy for other projects to etw included so this was mainly mainly a reason that they got successful and now it is included in G. unaltered which is called management system an open-source just part of common cold continent by the system and where you can do and GIS and SDI through the web and the prices w is now the default w server but also see can have included prices w it's is a a well-known projects and open source and open data world and that my next presentation is going to be is going to be on c can assign not going both from many here there was a project for a code of America open data catalog that use prices w now there's a new project called a type of up there is a city which is at the heart a project that uses and actually use it is the 1st thing you spices W 2 to version 2 and then we have some commercial projects like bomblets expert exchange with will which is also that built on top of dual prices w x HyperMap duty and user so Obama succeeds is based on wt uses its forces W 3 and they're all right now let's move to some well known implementations so that used to see prices w right now for 2 years or for 2 years of the data God is implemented using C can and prices WS that uh CW API has been working in great for 2 years old no problems and when we started it was like happening big datasets 2 dates 2 million that and some more recent developments like no 1 else uses it its and Department states and you can see it a nice marked with like deployments on our website so what's what's coming in the future who we are working on till the cat the peace support ElasticSearch solar and we're moving our testing to buy duct tests so far Karl Eugen 0 I have some because of all the duration of then would all have time so I'll just keep this 1 so so how can you get involved generates us through the
website CW we have a mailing list can subscribe to we are always of of free node on IRC and you can find this on due some ideas card the organization and also through was deal links are there and also you can find professional support in that was your registry and we have a workshop which was funded by Oracle called fossil marine data network which is also line and just not as last note we call them that we call named spices W and B with the name that because it doesn't in honor of that never never who who was the head of the use of the 3 Working Groups who passed away 3 years ago and they serve as he people status lot to do this W 3 of the very early stage so that it for me and questions thank you and also for this
nice presentation with time for 1 or 2 questions and is it possible to get data from prices w into local library clients like and belong to that kind of stuff like that and go to I mean a lot of my research of colleagues that side of of papers from the scientists but reality the data so I was wondering who would be very handy to get the data the metadata from is w into right tools like and blogs so that people can cite the that they're using in scientific papers more easily while they're being yeah but haven't tried that but you you you can actually I've got the ideas of the data sets and then a link to it URL because this is UW it within this year the database and there is an indentifier so if you just link to a depth that get feature by the and then you are able to link directly to the to the data is limited data record for for this dataset so it's possible I haven't tried it is a good idea now has had which have who would you recommend if I want to determinant of the term new that on 78 is who sees the review transaction to prices produce something which was which is compatible for users to think of that so that's a big discussion then we we we stay away from any you why so far we don't do any metadata editor because we wanted to be all that only the server side but there are so many implementations out of their that do of fools stock-and-bond examined like metadata editor I mean network is here and here the calendar editorials was his great we there is the there are the June old has an editor or account has had well metabolic has an editorial guides metal-rich or all so there there are it's it's like I cannot recommend 1 use everyone and try yourself and figure out and we we have we are testing everyone everything to be honest well thank you and if there are other questions so we might postpone them because most will have another talk in this session so and you have the possibility to ask later on