Bringing benthic data to the surface - moving Marine Recorder into an open source spatial database.

Video thumbnail (Frame 0) Video thumbnail (Frame 591) Video thumbnail (Frame 1824) Video thumbnail (Frame 2731) Video thumbnail (Frame 3290) Video thumbnail (Frame 3883) Video thumbnail (Frame 4425) Video thumbnail (Frame 4986) Video thumbnail (Frame 5621) Video thumbnail (Frame 6333) Video thumbnail (Frame 7034) Video thumbnail (Frame 7699) Video thumbnail (Frame 8662) Video thumbnail (Frame 10660) Video thumbnail (Frame 11672) Video thumbnail (Frame 12775) Video thumbnail (Frame 13343) Video thumbnail (Frame 13963) Video thumbnail (Frame 14919) Video thumbnail (Frame 15809) Video thumbnail (Frame 16463) Video thumbnail (Frame 16947) Video thumbnail (Frame 17417) Video thumbnail (Frame 18158) Video thumbnail (Frame 18620) Video thumbnail (Frame 19816) Video thumbnail (Frame 20507) Video thumbnail (Frame 21642) Video thumbnail (Frame 22115) Video thumbnail (Frame 23871) Video thumbnail (Frame 24766) Video thumbnail (Frame 26031) Video thumbnail (Frame 26864) Video thumbnail (Frame 28115) Video thumbnail (Frame 32668)
Video in TIB AV-Portal: Bringing benthic data to the surface - moving Marine Recorder into an open source spatial database.

Formal Metadata

Bringing benthic data to the surface - moving Marine Recorder into an open source spatial database.
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
The Joint Nature Conservation Committee (JNCC) is one of a number of organisations that run inshore and offshore surveys to collect data on seabed biodiversity (e.g. species records and habitat types). The main storage and collation system for these data is Marine Recorder, a Microsoft Access-based database application developed by JNCC in 2002. Despite several updates since then, the system is now significantly out of date, and the lack of internet connectivity or the ability to store spatial data is a major problem. As a result of these issues, organisations in the UK that collect benthic data are investigating more up-to-date systems and there is a significant risk of data fragmentation. JNCC are currently in the process of moving all of our spatial data from a file based system into PostGIS. This talk will describe JNCC’s open-source geospatial information strategy and how we are integrating Marine Recorder into this, discussing the efficiency savings that this provides, as well as our plans for developing a full open-source, online replacement for Marine Recorder in the future.
Computer animation Open source Meeting/Interview Surface Database Row (database)
Surface Open source Physical law Open source Database Staff (military) Numbering scheme Open set Computer animation Strategy game Conservation law Video game console Office suite Conservation law Local ring Reading (process) Row (database)
Standard deviation State of matter Water vapor Coordinate system Numbering scheme Discrete element method Special unitary group Repeating decimal Power (physics) Programmer (hardware) Computer animation Conservation law Species Summierbarkeit Game theory
Area Email Dependent and independent variables Decimal Time zone Mathematical analysis Numbering scheme Online help Line (geometry) Numbering scheme Local area network Special unitary group Area Component-based software engineering Data management Programmer (hardware) Computer animation Software Website Species Quicksort Conservation law Summierbarkeit
Area Type theory Computer animation Different (Kate Ryan album) Archaeological field survey Formal verification 1 (number) Website Selectivity (electronic) Conservation law
Pulse (signal processing) CAN bus Computer animation Natural number Multiplication sign Basis <Mathematik> Quicksort
Type theory Greatest element MUD IRIS-T Sampling (statistics) Self-organization Bit Quicksort Species Row (database)
Slide rule Digital photography Computer animation Multiplication sign Archaeological field survey Videoconferencing Species Conservation law Fehlererkennung
Run time (program lifecycle phase) Computer animation Software Term (mathematics) Maxima and minima Bit High-level programming language Conservation law Row (database)
Dependent and independent variables User interface Computer file Key (cryptography) Structural load Multiplication sign Debugger Mathematical analysis Archaeological field survey Database Bit Computer icon Demoscene Front and back ends Medical imaging Spreadsheet Computer animation Query language Species Data structure Quicksort Table (information)
Area User interface Gateway (telecommunications) Gateway (telecommunications) Computer file File format Database Database Frustration Shape (magazine) Area Computer animation Software Query language Website Species Conservation law Website
Computer animation Integrated development environment Multiplication sign File archiver Website Open set Conservation law Open set Reading (process) Attribute grammar
Standard deviation Dependent and independent variables Code Software developer Metadata Library catalog Library catalog Metadata Field (computer science) Product (business) Message passing Computer animation Moving average Computer-assisted translation
Scripting language Computer animation Validity (statistics) Link (knot theory) Projective plane Archaeological field survey Metadata Website Computer-assisted translation Metadata Physical system Formal language
Web portal Binomial coefficient Line (geometry) Archaeological field survey Point (geometry) Set (mathematics) Database Uniform resource locator Computer animation Species Conservation law Capability Maturity Model World Wide Web Consortium
Computer animation Vector space Computer file Self-organization Cuboid Set (mathematics) Software testing Shape (magazine) Datei-Server Reading (process) Electric current Physical system
Web page Server (computing) Presentation of a group Computer file State of matter Texture mapping Vector space Repository (publishing) File system Process (computing) Traffic reporting Physical system World Wide Web Consortium Projective plane Database 10 (number) Computer animation Raster graphics Repository (publishing) Strategy game Point cloud Object (grammar) Conservation law Electric current Row (database)
User interface Purchasing Raw image format Standard deviation Link (knot theory) Key (cryptography) Texture mapping Direction (geometry) Multiplication sign Device driver Database Bit Open set Coprocessor Metadata Formal language Subject indexing Computer animation Raster graphics Repository (publishing) Phase transition Strategy game Repository (publishing) World Wide Web Consortium
Filter <Stochastik> Point (geometry) Greatest element Link (knot theory) Computer file Sheaf (mathematics) Shape (magazine) Value-added network Web 2.0 Operator (mathematics) Cuboid Physical law Alpha (investment) Area Service (economics) Mapping View (database) Moment (mathematics) Database CAN bus Type theory Computer animation Interface (computing) Bridging (networking) Hill differential equation Species Conservation law Row (database) Wide area network
Run time (program lifecycle phase) Divisor Concentric Multiplication sign Interactive television Coroutine Shared memory Bit Database Web browser Replication (computing) Front and back ends Revision control Computer animation Computer configuration Hybrid computer Endliche Modelltheorie Data structure Conservation law Newton's law of universal gravitation
Computer file Open source Structural load Interface (computing) Multiplication sign Stress (mechanics) 1 (number) Shape (magazine) Revision control Message passing Computer animation Software Interface (computing) Moving average Object (grammar) Traffic reporting Sanitary sewer
Point (geometry) Implementation Statistics Computer file Range (statistics) Web browser Meeting/Interview Different (Kate Ryan album) Bridging (networking) Computer configuration Electronic visual display Office suite Physical system Area Interface (computing) Moment (mathematics) Projective plane Planning Database Bit Incidence algebra Word Computer animation Personal digital assistant Website Right angle Reading (process)
can I relate to introduce genes origins and non bringing than that that of the surface living marine recorder into an open source spatial database great thanks for coming and this is going a similar so the last similar
problem I think I'm sorry I James I work for the joint nature conservation pretty in the UK we have a government body the laws of nature conservation and this the problem is
talking about who we are what
we do you have a middle ground pieces of local marine recorder uh it's about are geospatial information sources say how we're changing how we handle that spatial data and how we can ensure include re-recorded that so Gezici uh is a constellation body and you can call a small agency about 130 staff across 2 officers um and we do the things so it's all about advising governments on consolation both the UK and internationally and a stroke about 3 so is a reading so I'm sure you all know that the
UK is size of into 4 separate countries um hands the UK government delegates power adds to the don't demonstrations of those countries and they also have their in nature conservation bodies that are responsible for nature conservation in this country so JCC as a kind of UK-wide remits that we try and provide some coordination between them and by doing things like assets setting common monitoring standards so when people go out so uh monitor the by the state of course species they have in the country and world measuring the same thing during its a joined-up picture for the whole of the UK uh we also surveillance
schemes for a particular species everything from birds that down some mosses and everything in between uh and these are actually the the seasons run by NGO partners we work with but uh we provide that any government funding that gets and we also give them some help with data management analysis and that sort of thing and then the because
what we do is marine conservation and so this is the U. K. 's Marine Protected area network um the individual country agencies responsible for uh the seabed at to 12 nautical miles which is the bottom line you can see just from the coast but we lead on the seabed in the offshore areas beyond that line and once we do things like uh sites
and selection so deciding what apostasy which should be protected um verification so checking the things that we think there are actually there and uh then monitoring so once you've done that there's a place area making sure that the protection that you have there is working and doing what you expect the of all those things you have to actually get on boats and go back to your site and that and wouldn't Howard
boats but we work with a few nations have ones so this 1 belongs to Sc fast which is like embodies the center for of the Environment Fisheries and aquaculture science the fuel of work with I'm so typically about like this regards on 2 or 3 week cruise uh to somewhere offshore and gather a few different types of data and so the
acoustic data this is sort of uh there are a few different uh falls below a working basis and ways this and now the pulse of sound and then pick up the occasion that bounces back of things and that the time it's a sound come back says we have far away a thing is so you get an idea of the debt so you can build up a kind of 3 D picture like on the far side of the nature of the can also tells you something about uh the the
kind of nature of the thing is bouncing off so you can tell whether it's a hard and substrates rock or whether soft like mad have their we also take
physical samples of the bottom uh so with grab sampling you drop a big piece of machinery like that 1 that and the bottom and has jaws and bring up a lot about like this and I get saved through various different size meshes uh which tells you a bit about the settlement types so whether it's sort of gravel or sand or mud and also
when you submit you find all the organisms living in the mud and you can send them off to be identified in the species records and then we also do you
camera surveys so this is a uh a camera slide show on the left we need to its along with that's behind about and error codes video which is mostly used to identify the kind of habitat um and also you'd still photographs you might take a still photograph every minute of your time and also if you see something interesting when you're watching
the the video on the monitor on the boat and those is used for identifying specific species and all that data
needs to be stored somewhere themselves were gone summary recorder um this is what looks like it's navigation that was developed in the early 2000 so it's quite a looks a bit dated now and so uh it's built on a Microsoft Access um so to
uh this is the the front and this runs in access 2003 runtimes to and fairly ancient and software terms and so this is where the
behind the scenes so has this back database that this is the front and there we have a sizable fish icon that somebody designed 1st in 2002 and it doesn't like any UK species of using but they are and then the back and called the NB and data file because the NBA and trust as a ninja responsible for uh kind of coordinating biodiversity
recording in the UK and the the data structure in that's back and follows a set up to match their data structures that we can share data with them and the image is quite a complicated data structure uh say then we also have the snapshot over here which is the sort of simplified structure um and that's produced by a snapshot tool which is just a database that has uh related queries that pulled back ends and put into that such a file that is designed to be used for easiest of querying them in the in the main back and you have um there really hundreds of tables with lots of foreign keys and things worse the snapshot is a bit simpler to actually go ahead and plants some data that you want to use for analysis and then there's some of other tools for emerging fast together different things and so is basically a big collection of access databases and that has a few issues is not all that easy to use uh um which means a lot from the data doesn't get uses motion should do you uh all the data entry has to divide front-ends player has some as printed in bold tools where there was really well and you end up when you're entering data dangerous survey quite often you end up reading if 1 spreadsheet and fighting it into the front ends which obviously isn't really efficient them and because there's a long time also means that we often subcontract steps because we don't have time themselves so that then costs of course load of money that
when all the way to use it so used by the country agencies as well and by some injuries to collects has seabed data the UK um and they each have their area uh and files over the will have multiple back and files and and roughly twice a year they each produce a snapshot of the common simplified format of that data and send them to us when we combine all of those into 1 UK-wide state-space uh I published on our website and also related to the gateway which is the uh a kind of web interface to the national biodiversity networks biodiversity data um
and we also some spatial layers of its which internal use and then also not as efficient they could be so it involves running queries in the database um and then usually saving there's a important endogenous and then converted to a shape file and so again some efficiency issues that emanate UK-wide database is actually a really valuable databases the the best the best dataset we have uh all of seabed species and habitats in the UK because of their fullest it doesn't get used read much should and so 1 of our main motivation for wanting to improve this software is uh these various frustrations issues that we have it's the fuel as well
so uh in June last year uh Elizabeth stresses that idea that she was the have Environment Secretary for the UK at the time she announced that death from which is the Department for the Environment Food and Rural Affairs so that the UK Environment parliament would be opening up their data archives of making 8 thousand datasets publicly available on this data log of the UK websites and
and there is a sense against the release primarily and this Open Government License which is basically equivalent to a Creative Commons Attribution License I note JCC is personalization we actually hold quite low data because of our UK-wide read it and so we've already been working on trying to get a handle on our own data resources and learn to deal with
developers have built so uh this which is a uh metadata catalog and useful top cats and uh it's inspire compliance has all the same fields required by the inspire metadata standards and and so this is the this indirectly looks like them all as a little aside all of the code for this is a freely available on GitHub and I would say excessive production thing that you could just take roll out uh but the code is available under a become a license so if you need a message catalog you free to take it from an adaptive here and
purposes um say this is that when we were given this challenge releasing him always open datasets this message catalog that we have been building became central to our uh response let's
um and we wanted to include marine recalled data within that and so we we use are quite logical general purpose scripting language and within JCC so we developed our scripts that would plants the metadata that we have in marine recorded it has some essays and then all everything required by inspires system estimates that about manually as well have been labeled as outstanding so uh that went into top cats which
projects you have everything you need the inspired standard and from there can be exported to validate dates stock of the UK websites uh pulled out of the data so break that down into datasets from individual surveys um and that is written since these extended hosted on the uh on our websites with a link from the data log of metadata and this is a big this is that this is all
this talk of the UK web portal looks like and so on these some of the marina coordinates dates etc. and then as this is is quite a big improvement in the kind of and discoverability of the data uh so this has searching you can find a set for species and also for a particular location that you don't have to
download access database and trying to convert it from the data you want but it's still not perfect democracy that has been split up so you don't have a whole UK dataset anymore that is anywhere since these are also it's not spatial data so still further improvements to be made um and
and other motivations for wanting to 3 recorded there is some work that we're doing Iran's changing the way that we handle spatial data as an organization as a whole and so this is what our special data uh guarantee looks like we have a set of file-based systems so the stopped the boxes are or network drives uh and they're mixes the shape files and as read you databases for vector data and then test for the rest of data that we have a set of reference datasets which is the top copies of data that we
use uh repeatedly and they have met state records in top uh then we have working files saved things that people using 4 different projects that are most depressing happens on people's desktops used object so sometimes you off and this is going to
get them so moving away from the file system uh to a spatial repository based on page present post just um there will be hosted in the Amazon cloud so with the current system that we have I don't know exactly how much they need tens of terabytes of spatial data and probably lots of which is repeats where people copied Elias user and project and so you it's not the most efficient system so moving out into a database and the cloud will be a big improvement and for people who want to make a matter for a report or something and they'll be able to get that data viajes server has certainly MSO WFS uh feed over there on the on the left your or yeah your left hand if people doing more heavy-duty
processing others direct taxes in the database as well with our and other tools and again we have talked about there's this kind of uh data discovery tools and so that has all the metadata and uh links into the spatial repository so we want to bring more in accordance this and isn't inherently spatial but it is a database and posters database so we thought we could just copy directly into their own and the the
1st whereas there was with ODBC Open database collected uh standards uh unfortunately access ODBC drivers only 32 bits and using 64 bit processor that just didn't work they would talk to each other so we turn to our general purpose scripting language which is on the pretty quite a good step in the end because uh in the course of time moving the data into posters you can also you back and learn the lesson Longfield's summary in accordance the proestrus phase gesture commentaries uh had 2 primary keys and spatial indexing and so on so that we have to have the Marine Corps data base replicates in a purchase and wanted some kind of user interface this is
our 1st attempt to that so this is very much a kind of alpha thing moment um but it does it does like um so we don't have a user requirements gathering exercise just incidentally that's fine that lots of people
would need section better use of this data that we have and a few things that came up for being able to you just get the um species or habitat records for a particular uh sites of particular protects the area um and also revealed to filter the individual species on individual habitat type and say this does that's it's got uh um this middle drop-down box you can select a particular protected area uh from all of the text in the UK offshore offshore so um and that's the pre operational pre-calculated querying the database and then that the species and filters down the bottom work using CQL filtering induced over them and this generates a web map with just the point uh for the text area that you want the species or habitat that you want them and that that web map includes a link to download shape file but also a new customer that WMS get capabilities link which you can paste into Q gyrus and the data points into the so that's
kind of the 1st steps toward making this way but this is where we ultimately want to be going so we've already got uh are both state-space model is just a direct replication of the structure that we had before knowing recorder and so we need to do some concentrations the other people who use inter-coder um to try and get a trial work at which bits that reaction Newton-John simplify structure of its and we also have need for um an offline version so some of the overseas Territories that we work with places like the Falklands and Helena and don't have reliable interactions all the time
so we're looking at spatial light as an offline database option for that file-based database uh and if anybody has any experience of a kind of hybrid online offline spatial databases and uh Satan in which are there would be very interested to hear about that and then this will have eventually a nice shiny browser front-ends to replace the year 2003 runtime version um there will be easier to use and uh it'll have nice automated so that dates import routines uh % divide our whole world some other methods that are so the data entry is quick and easy if we don't have to subcontract centigrade expense and then we'll have GeoServer has come already demonstrated data dissemination and sharing and so that's where aiming to get you and
1 other thing that came up for the challenge is have the hugest uh WMS and WFS interface and which is is quite nice if you're used to using that something before less technical users who use veil stress but shape file and drag it into objects OK Jess and it's a bit complicated trying to set up a collection and get a data that way so we're wondering about the possibility of developing a simplified version of that will be easier to use as well and so fast that it's against the that the take home message uh is a warning about developing your own
closed source software say this was developed uh initially having just the dancers user requirements um and that of people started using its own because it's closed source whenever it breaks which is quite often or whatever some the ones in future they come to us and we have to try and the thing it so we spend lots of time trying to fix this um and I hope is that with an open source version um hopefully we'll come back to you next year's conference probably more realistically in 2 or 3 years time when I signed a version of uh reports that when problems crop up at different people across the userbase failed to fix them uh that's the uh data gets used better uh and uh um the that is freely accessible and uses it should be so that's at any questions that it uh just for
gratification since you use are statistics and you mentioned a shiny browser fund and the user shiny package or is it just because it is that it was just that is that's the that's a good point and that is something that we're just beginning to look and see um I've done here deliver the reading about the possibility of using shiny as like on the front to to uh database I think that's not a of design purpose I think it may be done but the thing there probably better option that they asked them certainly the Chinese something that we're looking into and starting to use a little bit and uh it might well be something that we could use for data display on a web sites and that kind of thing there have a the of good enough answer for the question of how to create an interface to to 2 A. W. Richard WFS in the Netherlands we've got the pedia OK case it's uh data for holders of the Netherlands and in each of the diver for and created a plug-in to do just that and it's really easy it's just to read some files from the web site and it goes the sense show me after a big range said no problem things approach hi but as far as I understood that you're developing this new system for Europe let's main office before Holly's you like this implementation plan for Europe wage and those questions to Irish offices held there are like coming into this new world that yes area that's a good point the at the moment the way American words because it's all based on access is everybody has their own database files sitting on a large light somewhere in the office and then people end up e-mailing access databases to each other or something some kind of financial and incident minor and ultimately we like to have this online so here the main database will be in the Amazon Web Services probably um and ideally every then just don't have the front and all talking to the same uh at central database so there would be a need for sending files back and forth every 6 months the database will always have the full UK datasets uh they're ready to use so that's why we're going to be going here and they have a surprise that according to the stage we haven't uh done all that much talking to the uh different countries this is yet and so that's 1 thing but said yet with any of these projects that as you bring in more partners it becomes more challenging with lots of different requirements and so on and bridges into cross that so that's the plan in in the course the right what did you continuous Hutchinson