Logo TIB AV-Portal Logo TIB AV-Portal

Building Open Source Projects in Government Esri Ecosystems

Video in TIB AV-Portal: Building Open Source Projects in Government Esri Ecosystems

Formal Metadata

Building Open Source Projects in Government Esri Ecosystems
Title of Series
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Open Source Geospatial Foundation (OSGeo)
Production Year
Production Place
Portland, Oregon, United States of America

Content Metadata

Subject Area
The challenges that are most commonly discussed by proponents of open source in government technology relate to changing the culture among technical staff and explaining the value of open tools and systems. But beyond the political concerns and misperceptions, there are practical complications in implementing these tools inside proprietary tech ecosystems like Esri. Although it's becoming easier, injecting open source into the Esri stack can be convoluted, to say the least.For all of its challenges, however, there have been many successful open source implementations in all levels of government, from open data portals to full-scale applications. Using case studies from recent Code for America projects, this talk will identify some of the more difficult challenges and highlight a few techniques for integrating open source geo tools into the Esri stack with a focus on minimizing difficulty for the developer and maximizing benefit for the end user. The talk will focus on web applications and tools while touching on data interoperability, spatial analysis, and trainings/documentation.
Keywords esri government open source GIS civic technology
Projektive Geometrie Data management Computer animation Mapping Open source Code Multiplication sign
Web application Projektive Geometrie Focus (optics) Arithmetic mean Word Data management Process (computing) Computer animation Open source Code Different (Kate Ryan album)
Enterprise architecture Focus (optics) Arm Code Multiplication sign Open set Cartesian coordinate system Symbol table Projektive Geometrie Web application Computer animation Agreeableness Order (biology)
Web application Computer animation
Server (computing) Arm State of matter System administrator Software developer Data storage device Bit Instance (computer science) Front and back ends Word Data management Computer animation Quicksort Physical system
Web 2.0 Web application Programming language Medical imaging Server (computing) Computer animation Internetworking Cartesian coordinate system Formal language
Trail Projektive Geometrie Computer animation Software Quicksort
Projektive Geometrie Type theory Context awareness Computer animation Information Link (knot theory) Figurate number Mereology
Point (geometry) Web page Server (computing) Service (economics) Code Direction (geometry) Open set Information privacy Mereology Dimensional analysis Power (physics) Web 2.0 Projektive Geometrie Mechanism design Different (Kate Ryan album) Computer programming Information security Fingerprint Default (computer science) Octahedron Web portal Database Bit Set (mathematics) Cartesian coordinate system Flow separation Arithmetic mean Computer animation Website Quicksort
Dataflow Server (computing) Service (economics) Open source Mapping File format Transformation (genetics) Code Database Open set Cartesian coordinate system Run-time system Field (computer science) Web 2.0 Type theory Computer animation Software repository Different (Kate Ryan album) Order (biology) Right angle Quicksort
Uniform resource locator Email Service (economics) Permanent Computer animation Key (cryptography) Telecommunication Formal grammar Cartesian coordinate system
Slide rule Building Web portal Transformation (genetics) Block (periodic table) Structural load Open set Cartesian coordinate system Medical imaging Message passing Computer animation Formal grammar Data conversion
Mobile Web Trail Standard deviation Open source Mapping Information Computer file File format Digitizing Multiplication sign Bit Open set Shape (magazine) Mereology Cartesian coordinate system Fluid statics Projektive Geometrie Computer animation Data conversion Design of experiments
Type theory 4 (number) Projektive Geometrie Matching (graph theory) Computer animation Open source Elasticity (physics) Open set Cartesian coordinate system Address space Number Connected space
Implementation Open source Code Multiplication sign Data storage device Cartesian coordinate system Mereology Projektive Geometrie Data management Process (computing) Computer configuration Customer relationship management Physical system
Building Group action Momentum Mapping Open source Code Software developer Multiplication sign Mass Cartesian coordinate system Connected space Projektive Geometrie Web application Data management Computer animation Mixed reality Core dump Self-organization Quicksort Data conversion Local ring
Building Code Multiplication sign Source code Open set Parameter (computer programming) Mereology Optical disc drive Data conversion Physical system Web portal Linear regression Software developer Moment (mathematics) Maxima and minima Bit Representational state transfer Type theory Arithmetic mean Addressing mode Order (biology) Right angle Quicksort Point (geometry) Game controller Server (computing) Service (economics) Open source Amsterdam Ordnance Datum Web browser Element (mathematics) Product (business) Number Internetworking Metropolitan area network Form (programming) Engineering physics Time zone Standard deviation Information Forcing (mathematics) Physical law Bound state Line (geometry) Cartesian coordinate system System call Database normalization Word Computer animation Rewriting
my name is losing Diamond and I N a fellow at Code for America and I am 1 of 2 community managers of map time which will play you can talk about later we have things things to do now and in this talk is called uh building open-source projects in government as re ecosystems and there's some as revokes the room so the questions I can't answer I'm sure that they'll they'll be able answer them for you so this
talk is about building open-source web applications with government GIS data and there are a lot of different meanings to the word open source project and we could be talking about you know open source and GIS suffered and we could be talking about a whole bunch of other things but and we're gonna focus on today is an open source web applications because that's what we do a Code for America and done it this brings me on so really this process is
a 2 way street I'm a requires not only the civic technologists to do work but also the government employee whose managing data to do work on so whether you're from 1 side or the other and there are things for you but you so for those unfamiliar Code
for America is a nonprofit based in 7 Cisco and we work with local government to bring cities into the 21st our create 21st century cities which really means and try to implement I technology solutions and to government problems in the work actively with local governments and the Fellowship are you work directly with the specific cities of the Summer here the city of Lexington Kentucky and there are a few different tenets of a Code for America project and better pretty common had using Open Data Open Data is a big focus on the they um are mostly I'm constituent facing when applications although now Code for America projects include back and projects and city managed arm and the idea basically is to attack the problems with small tech solutions and so a lot of times the symbol to web application so I am as is clearly
the dominant vendor an enterprise GIS I I don't think anyone would disagree I'm and civic technologists in order to work with government in the geospace and need 1 have play nicely with it and there isn't going to be a time likely in the next few years where those that this is the change and so we need to do is learn how to how to work with as we technology so
let's talk about that the warrior talk about is of 4 basic steps to building an Open Source Web Application inside of an as we technology stack and step 1 is that we need to
understand the i ecosystem infrastructure basically what is that text act that they're working with on cities of various sizes use different types of as we technology and then you know sometimes is a combination of as renowned as rituals and just isn't as they were
before start talking about this I'm not a back-end developer I'm not a systems administrator I I don't I've never managed city government GIS before and this is all based on the research and knowledge and data so take it with a grain of salt and like verify the thing I say before you go and bring back to your city on I have however worked in
government and city and state so I have a little bit experience that on the so typically that we're talking about right now is hosted data and that 1 of the biggest challenges of GIS is actually managing geographic data and it's big it's hefty and has a lot of new wants to it and to store the geographic information on the S and GIS stands for systems so that up in data storage and data management and typically a city will and have an yes for server instance to host the data on and as this is a locally managed GIS server arm or word bill was the date on RTs online and both of these are a sort of I'm at the both as tools on server is more self-managed words are online is hosted on and we can get into the nuances of each of those and other time of but you can read all about them online in the different ways to get the idea is to figure out how the city you're working with is managing the data on
typically you know as I said could for America's making web applications after making a web application your application is going to communicate over the web to get access to data and and you know the web and regardless of what programming language you use it all speaks mostly the same language of http so we can communicate that way and plus you know people access things on the Internet mostly generally as saying these days so it makes your data more open anyway if you can access it on the web on there's a lot of
different ways that and all of this can be kind of strung together and inside of a city and this is an image from from RTS come out but detailing portal which is on the layer that sits in between and they can sit in between are just for server and online if you wanna pushed up to ArcGIS Online on so again you can get more into that of that put and online but the main
idea is like don't try to replace the city's infrastructure and we're just gonna like take it and work with it and so doing less is like them on track to so do less to us it on so once you sort of identified what that stack looks
like the next step is to identify the data that you're gonna need for your project I'm and it depends on what you're trying to do at home are you building a routing network are you trying to and update people when things are happening you know whatever kind of data you need you to identify what that data is so you can find out where it is on and there are some challenges to
this to on because certain types of data is protected the just open up in the on open it up online and hand you a link to it and maybe has personally-identifiable information and maybe the person who owns it is particularly territorial about it doesn't want put it online and and maybe it's stored in such a way that even if it was put online you when people access reuse it so these are questions that you have to ask when considering the data that you're using for your project and on eject here that a good civic tech project started in need of instead of starting with data it's best to kind of figure out a problem that you're trying to solve for a semi you're trying to serve instead of saying 0 I have this ability something interesting with that and so the problem 1st and then and then data this is the most important part is the 3rd
step which is enabling access to the data so you can you can get get the data you can't really make a project a so in the Esri context so that it can mean a couple
different things and we mentioned before dimension for how data can you be stored on on on Oct yes for server or arches online and through both of those mechanisms the owners of that data that can expose a REST service which is an an N points on the web that you can access via the web the HTTP and the it is not open by default and if you add power have data are just for server is close in only you can access it by by default same thing with RGS online you have dead set the permissions to say yes we all can access it and ends at 4 there's a lot of there's a few different ways that you can enable access for that but the point is that you have the this is the sort of government partner kind of role is to make sure that whoever's making the application has access to the data and and this is our probably the biggest challenge I would say with Code for America projects is getting access to data and because of privacy concerns I mean even on the RTS website poking through there's several pages when reading about exposing services about security and privacy in have a lock down your data and make sure that people on an access to it I'm so that's sort of the rhetoric of mind mean of shifting the rhetoric in the other direction of this is why we should open data and this is why you should have access to it is that so I think the way that we should go but but the point is is that you can and expose a service access that with an API an API is a way that databases and programs allow you to access certain parts and control the way that you access them and the talk too slow down a little bit on another way that Governments out
10 to sort of put data out there and publish data is through Open Data portals and maybe instead of exposing common API directly from re stuff on they pushed the data to an Open Data Portal like as a crater and or can come or archeus Open Data and which is new from us and directly integrated with RTS online so there's lots of different ways that you can put data out there so government are more if you're just running commentary tools you can put data out there and and then step forward is
and because ETL or an extract transform and load i'm and this is the idea of taking the data from that source shifting around so that it fits your application and then loading it into your application so this is what I
am a a REST service might look like at least the top of it and this is let's go get this data and as you know grab it have that's the extraction piece I'm if you're making if your application includes a web map you probably want to use TOJ signed and has a data format and end up with RTS right you don't you can't X on X for adjacency have to transform the data to duties on on RTs Open Data will have the duties on exports but right now if that make that transformation also maybe your application you know needs a specific fields or need to rename the field in order to interpret that data on so that's the transform piece and the 3rd piece is actually loaded into your application and and that is so a matter of just you know depending on your application whether it fully into a database server exporting to see is the some other way that your application is going to manage it on and then you know you repeat and depending on the different types of data that you need and or and if that data is updated then you can what kind of and keep that flow going between the data repository and application and so I have 3 Code for America example that sort
of show how this works and why it's valuable on the 1st 1 is an application called city c-gram grammar was I'm originally created by the Charlotte fellowship team I'm and is now being deployed in Seattle and Lexington as well and the i it say GO based on the her at location
based on in text messaging e-mail notifications about city services on including could enforcement violations building permits electrical permanent on and other datasets the key here is that you know without this kind of communication between where the data is being stored in host and GIS and the application like this with this is wouldn't be possible on forgive the gray
slash through this image but on city grammar has an ETL layer in the middle of it and cults by glass and was quite last does is it grabs data from the Open Data Portal or API and it does that transformation into city gram compliant duties on and then it loads intercity gram and so this layer in between allow is 1 for the application to Ryan and communicate with that of the Open Data Portal or an API constantly on and then you know the citizen can sign up for it and get a text message when there's a building permit within a block of the the house and such a and
another example is the eye open trails data converter and what this open trails let's next slide open trails
is a data standard on for trail data and trailhead data and parks stale eventually but and and the idea is that kind of enables parks to and moved their user experience design into the digital world and parks are really good at making people excited to be in parks in those maps and pamphlets and signage and information on but it's all sort of in the source static and parts are not very good at doing this online or mobile so the Open Data or the open trails data standard allows for the creation of applications by putting this data into a standard format the no open trails converter
itself is a tool for converting trail data into that format on this works a little bit differently than I ETA than us by glasses and the ETL layer for us a gram because it does require some manual work the city has to export a file and then upload that shape file to the converter and this kind of speaks to that situation of 0 1 0 1 0 exposes Cervantes unnatural to do that and wanna do that I only wanted do this 1 time and you can still convert data and use data in open source projects if it's stored and of a a GIS on the
3rd example is that the Lexington geocoder I and what this tool does is it allows you to type in an address and the does fuzzy matching on the address so that 322 W a number 4 th St matches 3 22 west of 4th Street written out which 1 is a big problem as you heard this morning Darryl talking about you coders and had occurs are 1 of the bigger problems that are left In the based on the geocoded just takes some open parcel data
and uses Elastic Search as awaits you fuzzy matching it's very small application and it uses this 1 dataset to enable the connections between all these other datasets and so again this is you GIS data being leverage in an open source project our and manage that way on kind of zipped through that really fast
but and those other times more projects examples on the Code for America get home new just a whole bunch about give hub so you can go in and play with that on there are a few other
considerations and they have to think about when building open-source projects the government on and 1 of them is hosting where is this application actually going to live on and there's a lot of other options for that and that we can talk about your eye also you have to keep in mind things like changing infrastructure and data storage on the part of the city if the city changes its system if the city updates or moves you know your application if it's automated may need to change as well the that's something to think about and also relationship management I have so many talks about open source and government are about how do you convince people to implement open source in government talk is not about that but it is super important and without that you kind of can can't really do anything you can start this process so relationship management is super super important and also
uh Code for America has local volunteer groups propagates and their goal is to work with the city on civic technology projects and they can be help a lot because they can establish relationships with the city on a and they can kind of have a give you sort of mass of people who all trip have different skills and can work together on to implement these projects I'm also there on is a pretty sweet organization comment time and and that map time groups are hands-on beginner Ferguson you have groups of 4 maps falling out with mapping geospatial and it's typically as Aaron matching earlier a mix of developers and GIS professionals who are marked there to learn from each other and kind of like this conference on so these groups can help a lot on this talk was about building
open-source web applications of government data and I think that we did that when it's really it's not as difficult I think is people make it seem and that I should say the impetus for this talk was last year at the code for America Summit 1 the GIS managers from 1 of the city's came up to me we started talking and he mentioned this team they built this really cool open-source application but I use in every stack so I can't take advantage of the school application because it exists outside of my ecosystem on and that is not true but there are many ways to connect and you know I will mention I've met in my intervention . troop before there are several tools that have been created by and so that is really R&D centers to connect that Esri core every stack ecosystem to open source tools and Kennedy that conversion so that's an that's pretty cool the new that on opening government data is a big deal it's really important sort of America works all the time and if you're doing that work then you're like totally awesome and told champion and at it's really worth it i'm because you get to enable all kinds of innovation and by doing making these data connections on all kinds people can only do this cool thing and then they totally candidates because you opened the data and then you feel really good and you can yourself on the back of story worth that a and I can it's
regressions I think if it I can you talk a little bit more about spy glass and how you identify sources and access those sources program and programmatically short on side in rights by glasses bounds by us was written by my colleague anyway land on his kind of a ruby genius I'm not a Ruby developers so I will be able to talk to technically about it but from what I understand spy glass on makes regular calls to but right now in Seattle for example on this is known as Rh example this is a Socratic example on makes regular calls to Secreta I'm and then it transforms and caches the data and that's another thing to mention actually is that I government servers are typically extremely slow on it really like crazy crazy slow and I mean unless you're talking about emergency data element that needs to be fast because it's emergency data and you an ambulance might want to have information quickly and so by doesn't actually caches the data so that the city grand when the remix calls by glass on it can return the cash the extremely quickly on the city ground holds the information about the fate you signed up within a quarter mile of your house city gram knows that and so Instagram asks for the information from spy glass it says get into the data that's in 0 a quarter mile from his house and he only cares about building permits and forces violations on and then that ways by glass doesn't have to do that work on and spyglass and zone excuse me city gram doesn't care about the data infrastructure and the idea is that city gram can can be built on top of any Open Data Portal use the rewrites by glass and it's kind of a template on and if all the code of course on line on the cut from if you get have so I can check out the question and How do you overcome the non-technical issues of getting into I mean like because we can all have suggest technical solutions to all these problems in getting the data how do you what have you seen in your experience is overcoming the non-technical obstacles in some a champion open data within government and you use of the word right there I championing on 1 of the big things is on to find someone who is a part of the system in government who can be a champion for your idea and typically and if you can and get someone on board on inside government that they can of they can provide that influence for your idea of I think that a lot of government IT people feel undervalued in like the work that they do isn't important there's a lot of people their hands tied you know would have no browsers earlier in some cities you know you can download any browser Versailles and you have to use IE 8 i'm because your city has services that rely on it so I am I think just being kind and respectful to the technical people in government I'm is also an and I really like the show not tell philosophy I think if you can build a minimum viable product and show really quickly the value of your application your idea and it's a lot harder to argue with then of rhetorical and argument at the end of but Waldo Jaquith at the US data open it has to he wrote a interesting by press recently about the sort of rhetoric we use talk about Open Data generally and and there's a lot of like you have to do this for you should do this a we paid for this and you need to open and I just doesn't work as well as like you're all the benefits to doing this so make sure not tell finding a champion being respectful and being positive are all great tactics used what question the person so I would follow over the whole 1 common baby so if I can I didn't say correctly that you and I will come to so it affected the data are published to our Argyros's Cyprus EPS services them anything of odd years for services so our 1st correctly that you consider it as a positive thinking should published through or where this vital services hopelessly by common would be my crew could be what support open so north so openly just a little stressful a yeah and then I mean that API eyes are great now bulk data is great anyway that you can access data is great but considering that the status quo is no data access and I think that any data access is better than no data access on them and once you kind of hit that point where rhetorically in in in the mind it's like Jack Open Data into it then you can start having conversations more about like what is ideal for certain types of users in applications and but yeah I think that that available data regardless of how it's available or are you aware of the 5 saucerful political sification phosphorus classification from before open data 5 far five-star sparsification opened it up here and it's basically tells this so 1st saw the main means that the star and the label somewhere on the Internet The 2nd star means so they are in some missions readable form of this through the source means uh that there is a standard for it for the false starts means of the data what we know they are they are linked you can you can always find do you make the world on the internet in in order to dull moment and five-star classified Open Data means that the 2 all linked between each other don't so was gold is way for invoking this so there's a question on if you could have 1 Open Data Spec API what would it be what is it the it said is that which is an open data standard what is it for due especially to home she sees some loss for sure the the we are in the in open source of law for and geospatial so for geospatial later the talking to both the history of argyria REST API than I would support to go for a user use the the people in this this was all of his yeah but just is use co redundant you know know they used to say the same about the really control coupled with which I just wanna say really quickly they also and to kind of you know in this moment of truth so tonight we're having a party nap time is having a party at also and there are a staggering number of hours the peas kind of terrifying but did really fire and we have rainbow stickers and stuff so you should come out as they apply at all yeah I wrote question moment this discussion were like really possibly considering of using the yes Try Open Data Data Portal whatever the hell politically that's really easy to make happen but I wanna know would somebody actually be able to use it in the open source community I mean is it useful yeah I start and I mean I like to talk to somebody who would use I however King County in Seattle Washington I don't know what to do that I'd like to talk to you out the man sitting right in front of you is the captain of the Code for America Boston became and you should talk him thing all can have really productive and wonderful conversations OK so let's