Managing public data on GitHub: Pay no attention to that git behind the curtain

Video in TIB AV-Portal: Managing public data on GitHub: Pay no attention to that git behind the curtain

Formal Metadata

Managing public data on GitHub: Pay no attention to that git behind the curtain
Title of Series
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Open Source Geospatial Foundation (OSGeo)
Production Year
Production Place
Portland, Oregon, United States of America

Content Metadata

Subject Area
The Atlanta Regional Commission (ARC) continuously solicits feedback on transportation data from local government partners. Historically, this process has taken the form of lots of markings on plotted maps with immeasurable amounts of manual work on the tail end to organize and interpret this feedback. Many tools developed specifically for this process today often fall short of the needs of agencies (such as geospatial presentation and tracking comments), yet the cost to develop or implement custom software is generally out of reach for government agencies.This presentation introduces a case study of the process to develop geospatial collaboration tools for managing transportation data directly hosted on GitHub pages (currently in development at and This approach was partially inspired by GitHub's recent features additions that make collaborating on geospatial data simple and elegant. Because these data span both functional and jurisdictional divisions, many of the greatest challenges have been project management related --- coordinating stakeholder feedback and project requirements. However, by utilizing the existing git/GitHub infrastructure, many of these requirements can be managed cost effectively. Moreover, the framework allows for direct integration with other application environments via the GitHub API and GDAL Tools, ensuring that local modifications to project data are committed back to the data repository.
Keywords github git open data government leaflet geojson transportation
Context awareness Presentation of a group Curvature Process (computing) Code Personal digital assistant Self-organization Bit Information privacy
Area Mathematics Computer animation Local ring Product (business)
Trail Functional (mathematics) Texture mapping Mapping State of matter View (database) Lattice (order) Mereology Word Process (computing) Computer animation Speech synthesis Data structure Figurate number Table (information) Associative property Social class Task (computing)
Dialect Process (computing) Computer animation State of matter System administrator Food energy
Degree (graph theory) Functional (mathematics) Dot product Mathematics Process (computing) Computer animation Software Interface (computing) Hierarchy Figurate number Social class
Web page Point (geometry) Area Trail Presentation of a group Open source Code Software developer Projective plane Continuum hypothesis Workstation <Musikinstrument> Expert system Shape (magazine) Web 2.0 Message passing Word Mathematics Computer animation Telecommunication Energy level Video game Right angle Control system
Web page Constraint (mathematics) Touchscreen Computer file Projective plane Diffuser (automotive) Resonator Motion capture Counting Staff (military) Shape (magazine) Drop (liquid) Revision control Web application Mathematics Computer animation Software Term (mathematics) Data conversion Condition number
Web page Presentation of a group Functional (mathematics) Multiplication sign Tape drive Online help Mathematics Boundary value problem Descriptive statistics Social class Domain name Area Email Focus (optics) Texture mapping Touchscreen Information Key (cryptography) Tesselation Structural load Electronic mailing list Sound effect Volume (thermodynamics) Cartesian coordinate system Data management Computer animation Table (information)
Overlay-Netz Texture mapping Computer file Tesselation File format Interface (computing) Projective plane Line (geometry) Shape (magazine) Event horizon Software bug Revision control Category of being Proof theory Mathematics Computer animation Software Different (Kate Ryan album) Cuboid Figurate number Condition number
Addition Email Dot product Validity (statistics) State of matter 1 (number) Principle of maximum entropy Counting Denial-of-service attack Cartesian coordinate system Mereology Number Wave packet Product (business) Frequency Mathematics Spur <Mathematik> Process (computing) Computer animation Bit rate Software Phase transition Local ring Resultant
Server (computing) Multiplication sign Source code 1 (number) Water vapor Open set Web browser Perspective (visual) Computer programming Frequency Mathematics Internetworking Monad (category theory) Software framework Addition Stapeldatei File format Projective plane Electronic mailing list Staff (military) Division (mathematics) Volume (thermodynamics) Instance (computer science) Cartesian coordinate system Entire function Process (computing) Computer animation Software Order (biology) Pattern language Window Local ring
Web page Axiom of choice Server (computing) Overhead (computing) Service (economics) Open source Computer file Code State of matter Multiplication sign Virtual machine Branch (computer science) Shape (magazine) Product (business) Attribute grammar Ideal (ethics) Cuboid Logic gate Associative property Area Overlay-Netz Dialect Texture mapping Information Tesselation File format Software developer Line (geometry) Cartesian coordinate system Type theory Uniform resource locator Process (computing) Integrated development environment Self-organization Right angle Cycle (graph theory) Geometry Library (computing)
so if you're here for the presentation just previous the mind you're lucky luck because I see the in the great a foundation for a for my talk and it's actually dovetails perfectly so so know that I don't know we were colluding you may be just the great organizers of phosphor yet unknown on and but my present had have by presentation is on a managing public data on get hub or pay no attention to that get behind the curtain and so it's and about and you know get having government and sometimes will unite at if you're a flat stock from leaflet earlier today talk a lot about experimentation creativity and I think that's on something that is really needed and the government sector and that's something that code for America's really stepping up and helping out with on but I think there's there's always room for that and so this presentation is really kind of looking at 1 case where I tried experiment a little bit would get out in managing this process n it went pretty well nothing exploded and so the context the but
I work at the while that's not the again
area I work at the and regional commission which is regional government to basically we build consensus with all the smaller local governments and counties and cities and we have a lot of counties in Georgia were notorious for that so there's a lot of partners to to mend relationships with and so that the story behind this this product was I each sensor so every 10 years and the urbanite urbanized areas are redrawn according to population change so people knew from Atlanta itself just go or you know always replaces and after we draw the various and along with that
of the federal highways this association asks for updated robe a functional classifications and unfamiliar with the OpenStreetMap but in an OSN speak at the function class cations just and you know that the highway tag and then any of those motorway trunk primary secondary will a residential but in the official UAE speech we have things like in the states of the freeways principle arterials you look you other classifications of basically in the same thing and then there's some funding tied to these classifications but as you is a weighted for the that's to keep track of all the roads that exist out there the
and so the the problem that we're facing was at in this process we like OK how are we gonna manage all the stakeholders always local governments from you know we have maps we can print out people can draw on get on the table and can do collaborative thing which is awesome and we have forms that people could fill out and I'm intangible Microsoft Word docs and haven't in a backer scan them and and we have a pile of paper to go through and in and basically have an analysts going and tag although from United's figure out which will be talking about the final map Taggart change all the 1st cations really can adopting task and you know so it's process that we do 1 manage and when I was in a meeting with the talked about was that this guy view better way to do it and I had a can demonstrate that the we like the 1st part but we don't like all the forms and having unstructured data and train it and data structures
and so you know if this is really kind of a huge process it's starts the anther highway administration which is on the national in scope communicating with all the deities state duties which G . is from the 1 that we kind of work with and then the energy that communicates with all the regions in the state and then all the regions kind of corny locally so it's is huge process and and there's no telling how many for ways to doing around the country and so
you know why not like as I mentioned OAS syntax earlier when I just user OSM from or their great tools for editing but which romance and that there was 1 of the you know really considering doing that or I was in my in my head and but the kind of motivations for not
using outweighed the degree reasons for using it to basically we have this said DOT road network that but we we really want to kind of work with an attach all those new outreach to and we wanted to have that on a custom interface pretty easily so we want to know and track changes with the function class cation numbering system and and we wanted to manage this whole approval process so that I you know you can filter of the hierarchy
and you know we 1st tried to kind of figure out what happened with the last census at 10 years ago but because this process happens every 10 years so you can imagine that the tools are going to be different the people are going to be different people probably just forgot how was done 10 years ago and we only know but this form was kind of a proposed as as a way to to actually Minister processing on about you but I would not wanna have a pilot with a stack of these forms in kind of going back and forth between a road network and doing all this manual entry
so about you know we the get I was kind of we use so why do we use it to and you talk a lot about some great tools that are out there that I'm I think kind of work inspired by ideas of kind of certain to render Jason and we have duties and I I 0 which is an amazing editor for and you know pulling up duties on our other from the shape data and then really easily easily editing it that's all I reminded of pages this great free hosting that you can go the also get help pages is really the curtain and this this talk right were kind of keeping get behind this this curtain that that is the update is just a web that's running on get have been taking advantage of the get of API and and and so I had to this really crucial golf because and sometimes when you're trying to convey an idea I just using words and really enough n by really quickly getting a demo up onto I get help pages you can convey a message like this is something that works and you know we could possibly use in this in this functional classification overall I mean issue tracking is another great thing will stick to give an edge of we have to corny with all these local governments and we want them all to be aware of what changes other people or proposing and so getting that integrated communications with the tracking issue-tracking was really important and we wanted to pass wishes with other agencies that's why I'm here today that's what it's all open source code and this is for can go mentality which I just caloric of seem to have come up with which is basically you know you put something up I get up and then get have really easily allows you for projects and make your own copy and to kind of customized for your own situation
so this is a continuum that I created from further justification for why we need a new this curtain around get which is kind of scary right now on the the left hand side we have familiarity with get and on the right hand side we have and you know your subsequent can relations relationship to so the 1st at the very top we have what's over control system from which you know I was at about a year and half ago and we have novice and expert in so it's kind of like a guess stages of grief or something and you have kind of indifference and then so you don't really care all get what what lies person you can but 1 that life normally what on you get and confusion might become really emission what's happening fear can discomfort reliance and you know your ultimately grow to really depend on get yeah so I'm kind of In this up a discomfort stage probably you know sometimes I i waiver depending on what I've done wrong on as far as committing to data that you have to get at bed through the originator get is probably had and the expert level but yeah the majority of of the target audience for this project is that the indifference station if you started talking about it it would lead to confusion most was likely 1 and so is a lot of battles you kind of have to overcome to D people comfortable with with this thing that I is really unfamiliar and so that's kind of you and there's been a lot of work to buy get help especially to kind of make get from permeate other other areas besides just suffer development I think that's amazing but I don't think were quite to the point where the average Joes can recognize what get the so the workflow for these projects
and so we had this G that have road network to we wanted to some listener changes to so we got that from them in shape version we did that over the ogre conversion to Du J. Sun and it's some the mean projection you me to that because there's a genome and earlier in that session last about a file size constraints forget had in terms of we actually had a splitter with a drop a lot of roads from the from the dataset and the Metasploit by county so to keep the file size nice and small because he's a huge road networks there were talking about then we look that up onto a debate pages web app which are shown a 2nd and we had government staff and interview the road segments and see with the existing conditions are in some change request for given a resonance in the jurisdiction it and so you know the other
reason we split up the side you that you Jason road network in the counties is because we we're dealing with multiple jurisdictions there was like concerned by NSA and for for roads that are on the the county bound here the cross hold for jurisdictions and we wanted your partners of feel comfortable that they were so many changes for only for their jurisdiction and on also kind of you know just kind of keep that collaboration alive and so 1 of things we had to do was split up the due dates and these counties so is kind of a multipurpose a reason for that we actually kind of see this is a screen capture from get of and it shows a few the different counties and we would basically created a by teams that correlated to counties and so all the cities or counties that corresponds to a given count uh county we would assignment to that team and that's how have a kind of had access to the data and change requests and
so therefore they do have open up this application so to be shown up
yet so this is the application we built its you can see it's running on this radial regional that do have that I 0 that's the get pages domain and and basically we have a map over here and it's got map OX tiles on underneath it and you can see this key is kind of Janke his screen size off on but we can think about some basic tile there's to entitle mail to help people kind of us he was out there so this is like you know really kind of messy but when you zoom in looks a little nicer but it's just showing up and all the different classifications we have and we have like in city boundaries this people to people know where they're they're working from home we have urbanized area that's not very helpful but people requested it but so when when you log in as a manager and all he basically assigned to from the county or collection of counties effect if the jurisdiction and Spencer so these are of Fulton gap counties and that's a date I have access to in reality if I go to the the get of an agent i want to you know create a pull request for any of the the data that's on that tool for on this and repository I can do that but are users cannot about to open up kind you know command line interface and tape and get committed did add and all that stuff so this is really the way they interact with the data and so you you can click on road segments this is just showing kind of generally foot functionality the and when you monarch suggest a change for the road segment of this form pops up where you know you give the name road because that's not really enough with that the DOT maintains you um you say you with the entire segment that's highlighted just to and from intersections you you can see the existing functional class you can see but if it has a volume so a traffic counts for the road you can see that the time and then you can kind of proposing new functional class and so on we also have kind of a list of all the proposed change that is better than no uh created by using that same form and so you can see that this is kind of like a data table it searchable try to make it as user-friendly as possible but these are basically just for familiar with get out their issues on get out that reduce using API to pull that information can a formatted in a way that's tumor familiar to people and when you view it loads that road segment can against all the description justification that the user submitted and an so it's really just kind of a wraparound get of have focus on this particular application so back to the presentation here check again so that's the out of you may have
seen when I pull up 1 in the issue boxes this you change status all these little buttons a year and I was just really thinking event of the other milestones feature on get of I set milestones for different from you know issues and FIL like bug fixes a feature requester you know this is she's going to be in what version 1 . no we just read out to that for some odd these road requests are they proved or they advancing our in review I'm so were can adapting them to have interface for purposes and then we also built this and another tool that that allows you to basically compare the eyes existing conditions which is that the map on the left with all of the different categories of issues and I am so right here I think that's I really really advancing so it shows all road segments that are advancing you can do the same kinds of tile overlays of kind of get a picture of what the whole network looks like a region-wide so we also in the pool data down
because know we there's going to be some some work on on the back into that you know figure out which requested a proof which did I know I can push down the line so we have an next 4 button just expert issues in this is the and and also if you wanted to actually work within like arc map which are agency has because it's not that you Jason format I can to do you know use desktop that get Harbaugh I get pull pull that down in re-projected into from change the format and the shape file so this is really about the project timeline
aware that the tail end of it and so on on the left were the y axis we have the number of change requests and on the x axis we have data so you can see that the 1st phase of product was really training now is pretty much for the month of May so we had an if you request put in some from an early stage that the bulk of the request actually came at the very end of this mission period surprise have you isn't a reminder e-mail in a flood of a and issues get pulled that very same day so you the the MaxEnt unity 7 in in a single day I was like the Thursday before the Friday up deadline nisi gap in the weekend rate and in the review period is kind of what during right now so it's kind of you know reviewing the whole network all the change the people made and and figuring out what makes sense and you know what doesn't like is its just like local residential road actually need to be upgraded to an inner state no it's it's actually a local road speed bumps not in there's in a answering that review process right now the so get them on friday and into the total change across we had was about a thousand which if we had manage that through a stack of paper forms and trying to correct for that all back in the spatial data that have been really challenging and this is
just a summary of change requests kind of kind of the look at but I'm basically it shows that in about a number of changes requested and we actually I can put this year because we had 20 submissions that were indicating no proposed functional classification so they they just didn't fill out the form all the way and and really that ended up being just a form validation we were doing checking like making that a mandatory field and so we had some errors as as a result of that and the other thing to point out here is we had a huge number of local roads that were from proposed to be upgraded and you know from on a dataset 1 that's fine we're like OK that's what is meant is that process is a lot of changes but that's OK but actually we found out later from the DOT that that actually means there's going to be a lot of additional data collection on their part because they have to go out and like you know to figure out what the roadway design is like you traffic counts and so comes up a lot of funding questions in challenges to
so the other kind of overall challenges from the big ones was source data we we had I use this dataset from the DOT but the army was unfamiliar to us so the segment divisions were really strange and so is a lot of figuring out what were actually working with and I mentioned earlier that we had a pool of roads out of the the network and in order for a day of the server properly that's because about 80 % somewhere around that and maybe a little less of the entire network local roads and so we reaction to pull that out and then some the we ended up just just having a kind of the the next classification out all the way up to an estate because reader monad 1 until the question of them all as local roads and so we would have people can send us into like batch requests to add a local roads which ended up being a lot of manual work I users and water of these users they're all government staff and I'm a lot of their agencies her are still running on my but you know pretty old Windows XP Internet Explorer and so on you know trying to get people to download the from the latest browsers to make this application work was was really challenging sometimes we did have to show the bones get out and special people make comments and so that would occasionally confuse people and and also the procrastination issue I mention the deadline of the all always change across coming in that was that person personal issues like trying to manage the whole process and constructing people had Opening of cation time and you know really that the volume change request continues to be a challenge because we have these additional data collection efforts that are required by the DOT and also just kind of figuring out what makes sense from a network but perspective like going through all those even in GIS format is challenging so moving forward we're continuing to
processes change requests and and really really the goal is to make a cohesive network but to make sense In reflects actual I travel patterns from the considering a framework for other datasets and for instance a chance fission improvement program which can a man is a list of projects that are in the hopper for the like 5 year period and and you know we'd love to hear ideas from others I think I was hearing where is working on his amazing and and hopefully others can experiment with this stuff but I think that's kind of the the crucial crucial theme experimenting and you know it worked for a process might our for other processes I'm but were always curious to hear from others n this is at the
awarding border awesome state to be in really have to be here but having to take any questions if just the there was a greater market was just curious and look at and fuzzy and how you're getting the information from getting back to the GIS like in URLs attributes coming over automatically yours or the edits being made in the GIS and this is just tracking them yeah so basically and we use the ideal hubs API to the to read the data from data pages and the right back to it so in that from the get pages branch we're writing data to that uh that file and then to get that back into like a GIS environment but and you just pull that all that data down and converted and the shape file and we can kind of work with not the yeah yeah were reduced just kind of like adding at tributes to the duties on some all those falling lines it all a she did hopefully help if a question so all of which he built a soap rosary we should book open source library called coop K OPB seen have its of we solve this problem that he was 8 it's no . at he did have repository of and conserve loudest feature services were shapefile something going live Mark MAP and pulled from from GEO from get harbor or vice versa so has a server they want was duties on you can apply that as well so that might help solve that keep getting did how but I wanna pull out into a GIS rather formats map downloaded converted under all by hand and things like that it was 1 of the problems we act is every as art map but some you know kind of getting did have installed on their machines and uh you know huge areas no there's lot of overhead with that like going to write you know that so and that's good enough you time I look like the whole review process was just couple months but how was how long the whole process of getting the infrastructure set up for this and so look back in my commit history but on a kind of got started but we're kind inform by G . that need to get started around think next February around and so as I can I think I have a solution for this 1 and was like well let's let's see it so those kind of a rapid development cycle and and then so kind of think started around like in the February and wrapped up and may be so have you gotten interest and this type of product from other city government through the government organizations so G outweighs there they're actually interested in the process of no they're not really at liberty to kind of impose any process on other regions in the state but I think in the future and no they had interest and possibly that using this for some iteration of that we have finally reached out to other infuse around the country but I'm I think that's going the next step in it's not really clear what that the best form is for that and I am speaking at the Association of an QIOs from that might October so I think that's really gonna be Eric her chance to share this more broadly with that community but I would guess so you mentioned used to map boxes for the map we saw the data that we saw inside of the gate of application was that your choice sir does get get help I agree directly with my someone that was that was our choice because and we really 1 in that kind of tile overlays in there and the easiest way to get that out was for us was through tile no into low those up to what's in that box so there's not any like direct in because like that code on on get pages is just you know a similar JavaScript thrown on there and so it was just easiest for us to get up and running I would say thank you for for some that