Towards pycsw 2.0 - project report

Video in TIB AV-Portal: Towards pycsw 2.0 - project report

Formal Metadata

Towards pycsw 2.0 - project report
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
pycsw is an OGC CSW server implementation written in Python and is an official OSGeo Project. pycsw implements clause 10 (HTTP protocol binding - Catalogue Services for the Web, CSW) of the OpenGIS Catalogue Service Implementation Specification, version 2.0.2 and the upcoming version 3.0. pycsw allows for the publishing and discovery of geospatial metadata, providing a standards-based metadata and catalogue component of spatial data infrastructures. The project is certified OGC Compliant, and is an OGC Reference Implementation. The project currently powers numerous high profile activities such as US, Integrated Ocean Observing System (IOOS), US National Geothermal Data System (NGDS), National Oceanic and Atmospheric Administration (NOAA), USGS Coastal and Marine Geoscience Portal, US Department of State, US Department of Interior, Greek National Open Data portal ( and the WMO World Ozone and Ultraviolet Radiation Data Centre (WOUDC). This presentation starts with a status report of the pycsw project, followed by an open question/answer session to give a chance to users to interact with members of the pycsw project team. This presentation will cover the main features and enhancements made to pycsw last year (1.10 series), the upcoming 2.0 release as well as the future direction of the project.
Meeting/Interview Source code MIDI Quicksort Metadata
Explosion Computer animation Euler angles Traffic reporting
Computer font Mathematics Computer animation Meeting/Interview Bit Computer font
Meta element Email Suite (music) Common Information Model (computing) Programmable read-only memory Execution unit File format Open set Client (computing) Mechanism design Personal digital assistant Repository (publishing) Row (database) Physical system Proper map Software developer Electronic mailing list Branch (computer science) Database transaction Web syndication Process (computing) Endliche Modelltheorie Point (geometry) Open source Dependent and independent variables Connectivity (graph theory) Disintegration Library catalog Architecture Latent heat Dublin Core Profil (magazine) Term (mathematics) Configuration space Implementation Communications protocol Computing platform World Wide Web Consortium Default (computer science) Standard deviation Focus (optics) Graph (mathematics) Suite (music) Server (computing) Core dump Client (computing) Cartesian coordinate system Limit (category theory) System call Software Function (mathematics) String (computer science) Spherical cap Revision control Utility software Data Encryption Standard Social class Library (computing) Standard deviation Gradient Multiplication sign 1 (number) Mereology Database transaction E-text Uniform resource locator Geometry Semiconductor memory Military operation Query language Extension (kinesiology) Presentation of a group Software bug Email File format Temporal logic Computer file Open source Open set User profile Type theory Time evolution Interface (computing) Website Configuration space Software testing Sinc function Row (database) Digital filter Server (computing) Implementation MUD Codierung <Programmierung> Line (geometry) Metadata Revision control Operator (mathematics) Integrated development environment Software testing Default (computer science) Operations research Addition Noise (electronics) Raw image format Dependent and independent variables Dialect Forcing (mathematics) Database Library catalog Predicate (grammar) CAN bus Computer animation Web service Computing platform Extension (kinesiology)
Presentation of a group Graph (mathematics) Texture mapping Multiplication sign Combinational logic Analogy Open set Function (mathematics) Data model Geometry Mechanism design Query language Cuboid Process (computing) Extension (kinesiology) Library (computing) Parsing Mapping Temporal logic Software developer Parameter (computer programming) Open set Mechanism design Category of being Representation (politics) Reduction of order Spacetime Point (geometry) Server (computing) Computer file Link (knot theory) Disintegration Maxima and minima Control flow Library catalog Electronic mailing list Continuous integration Content (media) Element (mathematics) Revision control Latent heat Profil (magazine) Computer worm Graph (mathematics) Magneto-optical drive Library catalog Transformation (genetics) Uniform resource locator Word Kernel (computing) Computer animation Function (mathematics) Revision control Cuboid Library (computing) Extension (kinesiology)
Euclidean vector Dependent and independent variables Computer-generated imagery Temporal logic Client (computing) Open set Function (mathematics) Metadata Architecture Geometry Goodness of fit Mechanism design Profil (magazine) Software Core dump Software testing Extension (kinesiology) Abstraction Computer architecture Computer-generated imagery Temporal logic Client (computing) Bit Price index Type theory Computer animation Function (mathematics) System on a chip Quicksort Resultant
Scripting language Source code Link (knot theory) Freeware Computer animation Price index Rule of inference
Presentation of a group Installation art Code Multiplication sign Database Open set Mereology Variance Web 2.0 Data management Geometry Type theory Information Physical system God Source code Software developer Open source Statistics Windows Registry Open set Data management Duality (mathematics) System programming Website Software testing Physical system Sinc function Server (computing) Mapping Content management system Open source Line (geometry) Disintegration Directory service Hypercube Emulation Revision control Integrated development environment Software testing Implementation Computer-assisted translation World Wide Web Consortium Default (computer science) Default (computer science) Standard deviation Server (computing) Forcing (mathematics) State of matter Expert system Library catalog CAN bus Embedded system Computer animation Web service Revision control Computing platform Video game Library (computing) Extension (kinesiology) Elasticity (physics)
Windows Registry Standard deviation Musical ensemble Server (computing) Presentation of a group Link (knot theory) Multiplication sign Electronic program guide Electronic mailing list Client (computing) Disk read-and-write head Metadata Machine vision Meeting/Interview Computer network Implementation Determinant Conditional-access module Source code Email Electronic mailing list Plastikkarte Database transaction Database Line (geometry) Set (mathematics) Wiki Computer animation Software Blog Self-organization Personal area network Right angle Text editor Data logger Spatial data infrastructure Freeware Local ring Library (computing)
Computer animation
but well I phase and where come to
this session I hope you enjoyed your coffee and on now well prepared for this session on metadata the and from our 1st speaker is so I'm just sort source and you will provide an overview of recent enhancements and made to apply
CSW please where command lots of people so that
will come over 1 this is a project report for fighters W and actually the
titles should be a bit in the change because prices W 2 is actually released now now but anyway we can this is the overview of the outline of the with today's talk we're going to
to go through some introductory in details so we're going to see the features what's new in the new
version of fighters W and then we're going to focus on downstream projects and future developments so 1st introduction so by the of w is an odyssey server implementation it's written in Python it's open source under the MIT license it runs on all platforms that support Python and a party so I also always has been and always your project since last year so we officially graduated so our goal from the beginning was to fully implement the Open OpenGIS catalog service specification and a lot of all the way we implemented some more stuff like OpenSearch recently and the main goal of spices W is to allow everyone to publish their geospatial data so that any other application can discover them through this year's API the project has been certified by OGC since the beginning and at some point 2 years ago we became a reference implementation of C is w version 2 . 0 . 2 this year actually 1 month ago the Odyssey released the W 3 justification and that the same day we really spices w tool which implements his W 3 specifications so this is a very short overview of the project history started to in 2010 by Tom Crowley this quiz and grew cattle in metadata I joined the team in 2011 and it was more widely announced and and then we had the release version 1 in 2011 than things to off from there we were included in the noise you alive we were then certified with to then we minus to get spices w deployed data data of golf the official US data portal and then we today we have the release of version 2 and the reference limitation from Odyssey for forces W 3 so our main focus was from the beginning to provide a very lightweight library that can actually plug into existing databases of metadata and to be able to be as lightweight as possible and I mean in in terms of configuration and memory usage and how fast it goes and etc it's very extensible you cannot be data formats was provided by the mechanism so you can write your own profile of fires or whatever as you you might have and but from day 1 we said that we're always going to be looking at the site test for all to see so are test happens through sites testing and are on the unit tests as we'll we're run every day so some of the features are not going to mention all of them because it's a very long list W 2 . 0 . 2 and W 3 is supported we are and how we can harvest that WMS WFS WCS and other CSW catalogs we implement the i zone with that the metadata application profiles the entities profile and we 2 years ago we implemented actually were we were the 1st implementation of OGC OpenSearch extensions and so we support transactional could capabilities and the figurative candles distributed searching and many of many more and so well I'll just go so that we are very focused on their own standards so In order to be able to be compliant with his w we also had to be introduced to implement some other standards so this is a list of standards that we support along the way some other standards so came up in there in the in the process like and nite and they said the firm and the as you an open search so these are the SEW operations that are mentioned in the W 3 and specification actually only existed in this year's w that 2 under without the and harvests 1 which is a new addition now so the server will ask for pitiable the client will last for kids capabilities from the server and then after it knows what what type of system that will be after he discovers the API he didn't actually ask for records or ask from to harvest the catalogs and stuff like that and you can also do transactions so so it's so what's new in prices W 2 mainly we were waiting for all this seem to release the the sea is W 3 specification in order for us to really spices W 2 and this happens because of this he take some time to review the documents and after the actual implementation happening is that it takes some time to to to release the the specifications so we were waiting to release our software to and so will pursue W 3 L better aligns with existing and new world notices standards like it supports now OWS common version tool and filtered version 2 it is highly based on all is made based very extensively based on open search an API now nowadays so as his W 3 and the open searches are coming together in many aspects and also although we has support for querying why it has to be a a temporal extents and distributed searching is much much easier for for their catalogs some more detail some other thing conference also includes a simpler achieve EPA API the role make metadata responses for get records by a the uh requests and also this year was missing for many many before long so long dialect it's DP status calls were not now properly used in the expected ensues W 3 and right so it's all so Odyssey are provided in the stomach of compliance test suite that was developed and actually follow with spices w 2 so while we were the implementing since W 3 that there the tests were also progressing along and we could help each other at some point and also there's another reference implementation so after a new version of tests was available we were able to just test our current status against that site testing and and figure out if if something is broken in our implementation or not so it it there this process can happen for of the year and set at some point this is the testing suite was finalized so we were actually the 1st ones who made in the mailing means that we fully past the compliance tests and well at the time of release and all this he now has to reference implementations will ready so what do some more the this involves the use W 3 now the default version of mind prices w when you ask the catalog it will default to see W 3 because this is what the specifically the what the specification we're still back work about compatible with the 2 0 0 2 version but you have to specify it in order to get an aversion to response so all this this new behavior of but actually brokering internally API and actually this was only 1 component of broken this these parts WCI and this is why we actually needed to release a new major version because it ain't broke the API and so on this is how it works now you just send me your
belly your horse name and last w to the server they're the new specification will not return narrative or was it something that everybody wanted that happens in instead it will send you back at capabilities documents and always is something that we all weighted and if
you don't specify version also it won't give you narrow had CW used to be very very picky about what to include in the in the URL in when you send a request so now if you don't specify the version you will get also 3 point 0 . 0 people with this document and then give space specify everything you get what you are requesting so this is something we expect so that also open service was a big thing for places W 2 the villages see also implemented profile for with Jewish in time extensions on top of the and w 3 c specification of open search and that this is a very light mechanism to query a catalog and using its STP gets so now we we support that actually we support that since version 1 . 10 which was sometime ago entities in the integrated in this W 3 also so basically OpenSearch profile gives you these 3 major categories of how to how to please the edge in a cuckold so you can just search through keywords or through bounding box or through time extent or in any combination of those so this can happen like in 1 1 greedy so here are some some examples for
example this is how you you can query the kernel for a bounding box it's it's much better than the previous version it was longer anyway so these are some some examples and this is a key word and bounding box request and we have links here for more more examples if you are interested to to test leans yourself the whole presentation is underprices w based so if you go to publications you'll find for for the 2016 there was another major things that was needed for for Python w was to support Python 3 because python is moving into Python 3 and it was needed to I had to be there so we implemented all we switch to to using future and 6 library in order or in order to be able to work with Python 2 in Python 3 at the same time and also we are not doing continuous integration through Travis on you have using 3 versions of of Python so we are pretty sure if something breaks or not when we are developing new new features but another nice starts to this release I think it is the better adjacent support but actually there's that the specification doesn't say anything about Jason but we used to be doing some mapping between what we what the XML elements where in XML file and uh we were exporting that 2 adjacent so that other developers would just use Jason would be able to work with it so now we know that we use in a very nice library called XML to data and then we managed to get the various compact output and better transformation of full of adjacent so this was before this will if you request prices doubled for Jason this is
what you got but then it is a bit awkward because repeats things like text and that it's it's and very XML like with Jason and now this is the same result which feels much more or much better for everyone who uses sensation so this is a this is a good thing edition foreign Jason people so are more features we are fly harvesting WMT we did some improvements in the XML output mechanisms we actually support this week's metadata profile because he would it it was contributed and also we have the better we have temporal extent support w mostly later harvesting which was not their previous releases so just sort of overview of that detector of how this works we have applied which implemented things like and indices and all the other stuff we have the CW core within our pull based and open search in history was different within the core court then we have the client school requests the metadata and this isn't a typical set up for the architecture was we are using as colostomy safely and Alex
MLW asleep and by-products of as dependencies over applies only any type of basic test a light as the meet supports and then we go we communicate with the external world through CGI or w z i and the party or whatever else you want to to
use the installation is very very easy to do actually we
have been included in beyond there are some discussions about all deceased my licensing which is not compatible to the to the uh the to the dead and rules but by CW has also reasonable and do all the there of course Python pockets indexed and then we're also and have and actually there's an easier way to in solid the if you use a script it will set up prices W 4 minutes
and if it takes 4 and a half minutes it's OK if you take standards not something is wrong but this this is something you can use for doing development and also by w has been always you
life since version 5 and 5 which is like 1 5 years ago so sorry so when the prices w being a library is very easy for other projects to etw included so this was mainly mainly a reason that they got successful and now it is included in G. unaltered which is called management system an open-source just part of common cold continent by the system and where you can do and GIS and SDI through the web and the prices w is now the default w server but also see can have included prices w it's is a a well-known projects and open source and open data world and that my next presentation is going to be is going to be on c can assign not going both from many here there was a project for a code of America open data catalog that use prices w now there's a new project called a type of up there is a city which is at the heart a project that uses and actually use it is the 1st thing you spices W 2 to version 2 and then we have some commercial projects like bomblets expert exchange with will which is also that built on top of dual prices w x HyperMap duty and user so Obama succeeds is based on wt uses its forces W 3 and they're all right now let's move to some well known implementations so that used to see prices w right now for 2 years or for 2 years of the data God is implemented using C can and prices WS that uh CW API has been working in great for 2 years old no problems and when we started it was like happening big datasets 2 dates 2 million that and some more recent developments like no 1 else uses it its and Department states and you can see it a nice marked with like deployments on our website so what's what's coming in the future who we are working on till the cat the peace support ElasticSearch solar and we're moving our testing to buy duct tests so far Karl Eugen 0 I have some because of all the duration of then would all have time so I'll just keep this 1 so so how can you get involved generates us through the
website CW we have a mailing list can subscribe to we are always of of free node on IRC and you can find this on due some ideas card the organization and also through was deal links are there and also you can find professional support in that was your registry and we have a workshop which was funded by Oracle called fossil marine data network which is also line and just not as last note we call them that we call named spices W and B with the name that because it doesn't in honor of that never never who who was the head of the use of the 3 Working Groups who passed away 3 years ago and they serve as he people status lot to do this W 3 of the very early stage so that it for me and questions thank you and also for this
nice presentation with time for 1 or 2 questions and is it possible to get data from prices w into local library clients like and belong to that kind of stuff like that and go to I mean a lot of my research of colleagues that side of of papers from the scientists but reality the data so I was wondering who would be very handy to get the data the metadata from is w into right tools like and blogs so that people can cite the that they're using in scientific papers more easily while they're being yeah but haven't tried that but you you you can actually I've got the ideas of the data sets and then a link to it URL because this is UW it within this year the database and there is an indentifier so if you just link to a depth that get feature by the and then you are able to link directly to the to the data is limited data record for for this dataset so it's possible I haven't tried it is a good idea now has had which have who would you recommend if I want to determinant of the term new that on 78 is who sees the review transaction to prices produce something which was which is compatible for users to think of that so that's a big discussion then we we we stay away from any you why so far we don't do any metadata editor because we wanted to be all that only the server side but there are so many implementations out of their that do of fools stock-and-bond examined like metadata editor I mean network is here and here the calendar editorials was his great we there is the there are the June old has an editor or account has had well metabolic has an editorial guides metal-rich or all so there there are it's it's like I cannot recommend 1 use everyone and try yourself and figure out and we we have we are testing everyone everything to be honest well thank you and if there are other questions so we might postpone them because most will have another talk in this session so and you have the possibility to ask later on