"Do This, and also That: Integrating Open Source tools into traditional GIS shops"
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 188 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/31606 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2014 | |
| Production Place | Portland, Oregon, United States of America |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G 2014 Portland163 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
Open sourcePoint (geometry)Stack (abstract data type)State of matterClient (computing)Wave packetINTEGRALStaff (military)IterationMathematicsCycle (graph theory)AdditionFocus (optics)Server (computing)Multiplication signFront and back endsBitData managementDatabaseGame theoryExtension (kinesiology)SoftwareGeometryProcess (computing)Enterprise architectureTotal S.A.1 (number)CuboidComputer fileProgrammer (hardware)Concurrency (computer science)Goodness of fitSoftware developerProduct (business)Different (Kate Ryan album)Physical systemTrailDisk read-and-write headUniverse (mathematics)AngleVideo gameWeb browserBlogComputer configurationWhiteboardInternetworkingElectronic program guidePolygonMereologyCartesian coordinate systemCore dumpMappingComputer programmingSmartphoneWeb 2.0Open setFile formatResultantLine (geometry)EmailShape (magazine)Service (economics)Row (database)Group actionLocal ringWeb applicationProjective planeSlide ruleLink (knot theory)InformationRevision controlMultiplicationText editorSinc functionExpert systemIP addressUser interfaceRelational databaseDatei-ServerStreaming mediaView (database)CodeData typeLevel (video gaming)Right angleArithmetic meanReal numberTask (computing)Representation (politics)Parameter (computer programming)Flash memoryNormal (geometry)Rule of inferenceFile viewerSpeech synthesisSubsetFreezingCASE <Informatik>Instance (computer science)Installation artNeuroinformatikSet (mathematics)System callWindowLibrary (computing)Mobile WebLatent heatBackupRaster graphicsThumbnailDiscrete groupReading (process)Semiconductor memoryQuantumDataflowQuicksortComputer chessPurchasingTerm (mathematics)DigitizingoutputContext awarenessHypermediaPower (physics)AreaPattern languageWritingSpring (hydrology)Field (computer science)Numerical taxonomyUniform resource locatorDesign by contractSelectivity (electronic)Endliche ModelltheoriePresentation of a groupArmFeasibility studyWebsiteIntegerAbsolute valueExtreme programmingSequelMultilaterationIntegrated development environmentMaxima and minimaFreewareDimensional analysisGraph (mathematics)Data integrityAddress spaceCategory of beingSpline (mathematics)Dot productMehrplatzsystemSurfaceInformation technology consultingWeightFlow separationGenderComputer animation
Transcript: English(auto-generated)
00:00
Good morning. Like he said, I'm Sarah. I hope you're in the right track that you intended to be in. I'm Sarah Safavi with a weird last name. Thank you all for coming out this morning. I'm kind of surprised at how many people are here because if I weren't talking I'd be listening to Vladimir right now and I wouldn't be here at all. So I want to talk a little bit, as you can see, about integrating open source into traditional GIS shops.
00:21
And what I mean by traditional, of course, is proprietary based GIS shops with different tech stacks that we all know and love. A little bit about me. I'm coming from... Hello? Oh, sorry about that. Let me... Is that better? Sorry, I can't hear it back here. So a little bit about me. I'm coming from a developer standpoint. I am a programmer.
00:42
I write code for food. I've worked in lots of different shops, public sector, private sector, some GIS specific shops, some not GIS, and I'm the only person doing programming at GIS. So this talk's based on a lot of experiences that I've had in some of these different shops. A little bit, of course, dramatized for the slides.
01:01
But basically this is from real life situations of shops I was in where there was an instance or a use case of wanting to integrate some open source software and what we did or what we could have done in some cases. So, hold on a second. There we go. Just so we are coming from a common standpoint, for the sake of my talk, this is what I mean
01:22
when I say open source. Everybody here is probably totally cool with that, but sometimes there's a little confusion. So, I mean, free software that's freely available. Y'all probably know what freeze and beer, freeze and speech is referring to. I just want to make sure there's some ground rules here. So say you're a GIS manager and you're interested in open source software for your shop,
01:41
but maybe you are anticipating a little pushback from your users or maybe your IT department. Yeah, I see a lot of heads nodding. Yeah. Or maybe you're on the other side. You're one of the GIS professionals in your shop, but you report to a GIS manager who's a little bit old school maybe or standoffish, a little intimidated by the idea of GIS. Or maybe you run an IT department the same way who is telling you, no, we can't do open source or something.
02:03
Basically, no matter what angle you're coming from, probably a lot of you are going to run into the same obstacles that I've seen in real life that we've all seen when you start talking open source in a traditional GIS shop. Most shops, like I said, are proprietary. Most of your tech stacks are going to be, you know, who?
02:20
These, Esri, Intergraph, whatever. You're going to be working with people who have used that software for a long time, decades, and they're not really open to suddenly switching. Switching everything all at once to open source is not really a great business idea generally. You're going to throw a lot of road bumps at your management, throw a lot of road bumps at your IT and your users. Nobody wants to see that. And also the idea of open source to people who are not really tech savvy maybe,
02:45
or not really familiar with the concept can be intimidating or scary. They're like, that's that open source stuff. It's for programmers, not for me. It's not for users. So this is stuff that I've heard a lot of people saying in real life, and you've probably heard it too. But one thing that these users or managers or whoever your naysayers might be don't really realize is something we all take
03:04
for granted, generally all of us here in this room. Open source is sneaky, and you're using it all the time, and you just don't realize it. So what I like to do is sort of throw this at those naysayers and be like, actually, you already use open source. It's not so scary. There's tons of it in use today. You are a regular user.
03:20
You're not a programmer. You don't have to be scared by it because you probably use Firefox in your browser. You probably use 7-Zip or maybe your blogs on WordPress or something. That's open source software, and you're a user. So this is like obvious stuff to a lot of us here in this room, but a lot of times we forget this obvious stuff is not so obvious to our users or our managers or even our IT department, depending on where you are, because especially in the public sector,
03:43
a lot of IT departments are a little unfamiliar. So hammering on that point that it's actually not scary, it's not intimidating, and it's something you already use can be really good, and then you bring that back around to your GIS shop and say, it's just as easy. You want to integrate a little bit of open source GIS into your existing GIS shop. It's like telling everybody to use Firefox instead of Internet Explorer.
04:03
It's not really that different, and it can be just as productive. Firefox is a lot nicer than Internet Explorer, generally speaking, and, of course, you've got all of these advantages that we all know and love about open source. Management really loves to hear about saving money. Saving money on license fees or ELA, stuff like that, obviously, that's a good thing,
04:24
so they always like hearing that. It's nice also, your technical people generally like to hear about flexibility. No vendor lock-in, of course, if you're not using proprietary data types or sources or proprietary stacks. It's easier to integrate with other brand stacks, which is especially good
04:41
if you're a shop that does a lot of, maybe you have a lot of clients or you do a lot of integral work with other agencies of your public sector, other companies, if you're a consultant, something like that. Not everybody is using the same tech stack, so it's nice to use open standards so that you can more freely move between those different stacks. Sometimes also these are semi-controversial points, but it's also fun to think about.
05:03
Advantages of using open source include keeping up with the G.O. Joneses. If you're using open source software, generally it's on a lot faster iteration cycle, so new developments, new technologies, new innovations will make it to your users, make it to your consumers a lot faster than a proprietary stack that has like a six month or one year or two year cycle where stuff is really set
05:23
in stone and slow moving, so that's really cool. People like seeing the new shiny when you're trying to promote open source at them. Also, I like telling people that if you move the focus away from the tools that they're used to, the tools that they're using now, you can stop thinking about the tools itself and start thinking about the work you're doing.
05:43
Stop thinking about the buttons you're clicking. Think more about the GIS that you want to do. A lot of times we don't realize how we're artificially constraining our GIS by being stuck with specific tools just because this is what we've always used. We don't think about other ways we could make solutions, other problems we could solve because we don't realize that thinking outside the tools is an option.
06:04
So the point of this talk is to tell you a little bit about how you can find that middle ground in your shops, whether you're a manager or user or you're somewhere in between or somewhere outside the GIS shop. You don't need to worry about switching everything at once. It doesn't have to be an all or nothing game.
06:21
Basically, integrating one piece, two piece of open source software at a time into your GIS shop can be something that's completely feasible, solves real problems and makes actual solutions happen and saves you a lot of time and money generally in the process. So the three points I want to go into, these are three examples, like I said,
06:42
drawn from jobs I've worked, places I've worked, clients I've had, of actual problems that were solved by talking about bringing in open source solutions because we didn't already have solutions in place for these problems. And each one's a little bit different use case. Each one has a little bit different outcome, but in the end, in all three cases,
07:01
we had totally workable, totally good solutions. Everybody was happy across the board from users to managers to tech. And like I say on there, yeah, I do mention Esri here because all three of the examples are based in Esri, but obviously you could be talking about anybody. So first example, so this use case or this real world scenario,
07:21
we had a team using Oracle Spatial Database behind Esri ArcGIS server and they had ArcGIS map services in front of it. And all of their users, they were a small mid-sized shop, all their users were desktop consumers. So they just had ArcGIS map services coming into ArcGIS for desktop. So this worked fine for them. They had some users who were not really GIS people, but they had enough ArcGIS licenses
07:44
to go around that they were just consuming in the desktop application, which most of you are probably familiar with. But they had some use cases popping up where they had public facing people that were not GIS people that were not part of their company who needed to be able to access the data in a read only format.
08:01
Obvious solution for this is web map. Keep in mind, this was a couple years ago when not everybody was talking web maps. It wasn't the immediate solution that popped into your head. And that's still the case in a lot of places you're going to find especially small companies in public sector. So they wanted to make some web maps. They weren't really sure how to do it. Management had the brilliant idea that everybody has smart phones.
08:21
We should be able to see these maps on our phones too because then we're totally cool and the kids will like us. So the obvious solution that probably presents itself immediately to you guys here is Leaflet. But at the time for a shop that had never done any web mapping, well they go to their account representative with Esri and they say like, what do we do?
08:41
And Esri says, well, we don't really think your ELA includes ArcGIS online credits. We're not sure how much it costs. But we can work up a quote for you and we'll get back to you in a couple weeks. And they also present a couple options like, well, if you don't want to use ArcGIS online, we have some SDKs like for Flex or Flash or Silverlight back in the day. And this seemed a little bit too crazy, too heavyweight for this shop
09:02
because they didn't have any like major developers. They only had like a couple layers they want to throw out for the public to see. They don't need something so heavy like the Flex Viewer. And they didn't really want to deal with another quote. They had to get past budget approval and everything for ArcGIS online credits since they only had like very small subset of public people who needed to see this data.
09:21
So Leaflet. Leaflet is an obvious solution for this. Just in case you're not familiar with Leaflet already, it's a JavaScript mapping API. Just like the ArcGIS for JavaScript mapping API. I don't remember what it's called. They have a JavaScript API. Google Maps everybody's familiar with is a JavaScript API. Open layers, of course.
09:42
But Leaflet is a very lightweight one and it's especially easy to pick up if you're not really a developer. You just want to make a basic map and throw some layers on it. It's super fast to spin up. It has really great documentation, huge growing user base. So there's a lot of support from the community
10:00
and there's a whole lot of widespread adoption. So everybody can answer your questions about Leaflet if you have issues. Because it's just like all these other mapping libraries that uses JavaScript, it's not a new technology that they would have necessarily had to learn. If they had gone with the ArcGIS online solution, of course they're not necessarily writing JavaScript but they're also constrained to what AGO gives them.
10:23
So they wanted something that gave them a little bit of future flexibility because they anticipated needs changing but they didn't want to invest too much time into it. So Leaflet was good for that. Also it met their mobile mapping needs because Leaflet makes great mobile maps. So you write one map, you view it in your browser, view it on your phone, works just fine. It's really lightweight.
10:41
So the integration advantage that worked for these guys is because they were already producing ArcGIS web mapping services. I forget the long names. They were already producing these web mapping services and consuming them on the desktop. Leaflet has built-in integration. It can consume those natively. It doesn't even need a plug-in. So it was literally I think like one line of code to add the layer
11:02
and like four lines of code to set up your map. So you don't even need to know how to code. You just copy and paste from the website. So what they got out of this was a new stack, Oracle Spatial Database still behind it. Everything's the same, ArcGIS server, ArcGIS map services and they just stuck Leaflet right in front as a consumer of those web mapping services, pushed that out to their public users
11:22
and everybody could see the map layers. It cost zero dollars because Leaflet is free and open source. It took minimal time to develop the new maps about what it would have taken on any other tech stack that they decided to go with. They didn't have to wait on their, as your account manager, to come back with a quote for AGO credits and that's awesome. And happy management, happy users,
11:41
management was super happy to see maps on his smartphone. He was like, yeah, this is great. And so that's a really easy solution to sell people on. Usually, shops that don't already have web mapping aren't really sure what they want it for but they think it's like something that they need. And they want to be able to spin it up fast, see results fast and then maybe iterate on it down the line. So using something like Leaflet that lets you do that
12:02
and then continually have completely open modification ability is really nice because you can get immediate web maps in literally five minutes and then if you bring in a developer or you have a developer that later wants to work on it, it's a great stack. So, by the way, if you're interested in Leaflet,
12:22
do what I'm going to do and go find Vladimir's talk that he's giving right now on the live stream that they're recording later because it's probably going to be super awesome. So, second example, multi-user geodatabase. This group was a really small team that was growing. It started just one guy doing GIS for everybody and then they added another person, add another person.
12:41
Lots of us are familiar with stuff like this. So they were on file geodatabases. They had a couple of them just living on a local file server. They all used ArcGIS desktop and they all consumed the same file geodatabases. And they had increasing issues with concurrent editing, data integrity, locks, all that fun stuff.
13:02
Their Esri ELA included an ArcGIS server license but they weren't really sure what to do with it. They just sort of like had the CD in plastic still and hadn't even installed it anywhere yet. But they decided that they needed to, you know, grow, get something as we threw around the phrase enterprise geodatabase and they're like, we need that, whatever that is, how do we do it? But we don't have any money.
13:21
They'd never used ArcGIS server but they did have it already. So an obvious solution that can be presented that costs them no additional money is set up post GIS which you can hear a lot about in about an hour or half an hour. But post GIS was a free solution that totally met their needs. It's a, for those who aren't familiar,
13:40
it's a spatially enabled relational database management system. So it's in the same universe as Oracle Spatial, SQL Server Spatial and all those other things. But the nice thing that it did is it gave them the ability to have multi-user editing, all this nice data integrity, issues are solved and they could move into a larger, more robust GIS shop
14:02
that supports additional users coming in. So post GIS is supported by like every GIS software that I looked into. Paul can tell you if there's anybody that doesn't support it but I don't think there is. Yeah, everybody supports it. So it was like zero change for the GIS users. They didn't even notice a difference except that now they're connecting to a database
14:20
that's fed at them from ArcGIS server instead of a file data database and that's like all they noticed. It's very easy to set up. There's great documentation and support. You can, there's an option to pay for support if you have nobody to set it up. I feel like I'm talking all about you guys now which is weird. But it's very easy to get management to get on board with this because if you're familiar with things like licensing
14:44
or ELAs with Oracle or Microsoft or SQL Server, we're talking about like, I don't know anything about it but the quotes I was getting was like 25,000 or 40,000 a year for very small shops. And for a shop that had like three GIS guides and is basically somebody's business started in his garage and was slowly growing, they just couldn't even fathom something like that.
15:02
So it was completely out of the realm of their consideration. So because of the cost savings, this was huge, easy to sell them on. They were a little worried about the idea that, oh, it's scary, we're setting up a enterprise geodatabase, we don't know anything about it. Also, we don't know what to do with ArcGIS Server. So that was like a point of intimidation for this shop
15:21
or a little fear point but honestly, it would have been the same with any solution that they went with because they were transitioning their workflow from one state to another and regardless of what's the technical solution they picked was, they were still gonna have training needs and not new staffing needs but new staff training needs, workflow changes and so on.
15:43
So really the cost for this in the end, zero dollars of course because PostGIS is free and open source. Some additional training, there's great documentation for PostGIS online and even Esri, when they were setting up ArcGIS Server has good documentation on the Esri docs on how to configure it with PostGIS backend with the ArcGIS Server
16:01
and they realize it's actually not that big at all at all because there's like the web interface for ArcGIS Server and they could just click some buttons and it was pretty easy. They had an IT team who did the actual database setup for them and just told them like here's the IP address and you don't need to think about anything else. So that's basically the extent of their efforts here. They realize it wasn't so bad in the end.
16:22
They saved themselves thousands and thousands and thousands of dollars and in the end they didn't have any more issues with data integrity. They had their multi-year geo database and it was awesome. So in an Esri stack, PostGIS can work happily with ArcGIS Server. It gives you an enterprise geo database, whatever you mean by that in your own shop
16:41
and it doesn't really cause any speed bumps for users because they don't even notice what's happening in the backend. So the new stack for these guys, like I said, PostGIS, ArcGIS Server, consuming in an ArcGIS desktop. If they wanted to make web apps or whatever down the road, they have that total ability of course. Cost them no money. It was safe, it was reliable.
17:01
Their data is no longer in danger, sitting on a box under somebody's desk that could be kicked and everybody's using the same file. They also had the ability to have all those nice things like automated backups and all that. So the last example, a little more complex than the last two. But again, great solution on the open source side.
17:22
So in this shop, we had a shop that was a public sector, had a manager, two GIS analysts and three concurrent use Esri ArcGIS desktop licenses. They wanted to get a couple interns in the summer to come digitize a bunch of old raster data. If any of you have worked with concurrent use licenses, you know that sometimes you get into that whole
17:40
and you yell down the hallway, get somebody to turn off their license so you can use it because all licenses are checked out. So this was a problem this shop was running into. Originally they were thinking, okay, we've got three licenses. Our two analysts can have them. Manager's not gonna do any GIS all summer and the interns have to share one license. That still is kind of bad because the interns are there on the same day, same hour and they're just trading back and forth
18:01
and also the manager can't like open ArcGIS to check something anytime he needs to. At the same time though, public agency, no budget and summertime towards the end of the fiscal, nobody's gonna improve an additional license on the ELA. So what they needed was a solution that was temporary so they can temporarily scale up their GIS capability,
18:22
let those interns get work done for them since they're there to get job experience, work experience in a real shop so you don't wanna just have them twiddling their thumbs and also get your digitizing done that's been sitting there for five years because nobody's had time to get to it. So the solution for this, of course, we told them about QGIS, QGIS for desktop.
18:41
Most of y'all probably have heard of it. There's great sessions on it today. In fact, I think there's one after me right here. Free, fully featured, open source desktop GIS software. What I tell people is it's same but different from what they're used to. Usually they're used to ArcGIS. QGIS is not a different category of product.
19:01
It's just the tools have different names or the buttons are in different places and the workflow might be different. But it does the same work that you need it to do. Of course, you guys, I don't have to convince you guys here but this is the argument you're gonna have with people, they think that it's a lesser product. The reason this is so great for a shop like this scenario is it's free, of course.
19:21
It's very easy to scale up temporarily and then come back down. Using it for a discrete subset of job tasks like this digitizing task where they're only doing one thing really minimizes your changing work habits and muscle memory and your training needs because it takes five minutes to learn how to digitize on QGIS and then you just let your interns do it all summer and it's great.
19:41
And of course, what I told these guys and what I tell people is that doesn't mean QGIS is for newbies. Doesn't mean it's like beginner GIS and if you're a real GIS person, you don't use GIS. It just means that using it when you're focusing on these small, discrete tasks easier because people aren't so intimidated by learning the entire product at once.
20:02
So the advantage you got here is you get to scale up, delegate some tasks to this. You're not paying for concurrent use licenses that you're only gonna use for like six weeks and you get all this work done, it's great. The interns get experience, they get more experience than they got in their school education because they'd actually never seen QGIS before and they're like, this is awesome. I can actually go home and do GIS.
20:21
I don't have to go to campus. That blew people's minds and that should be something we're changing but that's a different religious argument I have. So the new setup they had, they kept their ArcGIS licenses for their analysts and their manager to use. They didn't get rid of that. They kept their normal work going. They had products that they had to get out all the time so they couldn't change their workflows.
20:40
But they added QGIS installations to the intern computers. Everybody ended up learning together how to digitize on QGIS. So it increased the skill set of the entire shop even though just the interns were doing this work. It took maybe an hour of training altogether and most of that was figuring out how to install the software on Windows. Cost them no money, they got their work done, they scaled up temporarily
21:01
and they could be super flexible on their needs. No fighting for licenses which was awesome. So the bottom line that all of these three examples are getting at is integrating open source software can be a great solution to many problems in existing GIS shops. No need to take a huge plunge and change everything at once. It's very scary for people to hear open source
21:22
and think that means I have to change everything and everything I've learned for the last 15 years is now obsolete. They don't wanna hear that and that's not what we should be telling them. What we should be telling them is it's not that big a deal. It's like installing a new browser or a new application on your computer. It's learning one more thing that can increase your skill set, make your job and your shop more productive
21:41
and make your capability increase. So that's it for me. Hopefully I'm within time. Slides are there if you're interested in seeing them. It has links to the three things I talked about, leaflet, QGIS, PostGIS. You guys probably already all know how to find that. Please send me questions if you've got them. Contact info there.
22:05
Thank you for being on time. So we have plenty of time for questions, yeah. Yes. Use the mic for the recording. So how do you handle versioned editing? Versioned editing in PostGIS? Yes, multiple editors editing the same data.
22:22
So it's been a while since I've actually done it. I know it's possible. I don't wanna speak to telling you to look at the docs because I've got an expert here in the room. But what I would do is I'd have to go look at the docs to remember how to do that. Anybody else?
22:40
Oh, my one question I couldn't answer. I'm sorry. Stand up too if that helps anybody else. Have you, when you did the editing with QGIS, did you deal with complex features, not just connected dots, but lines, arcs, splines? So they were doing, a lot of what they were doing was like parking lots from aerials.
23:02
So they were mostly simple polygons. So that's part of why it was so easy for the training to come up. One other question, third-party applications. Many times, third-party applications are much more expensive than just the core GIS program.
23:21
So integrating third-party and getting third-party vendors to recognize that they have a stake in opening up their data. That's true. Yeah, I don't know if you have any specific examples you wanna speak to, but yeah, that's definitely a true point. Yeah, yeah, a lot of people forget about those.
23:48
Have you had any success talking to them about more open formats like Spatialite or Geopackage? Specifically, Spatialite, in a different use case, I was showing people how Spatialite
24:00
is really easy to pass around because they were trying to email shape files or shape folders at each other. And so I showed them Spatialite, and I think ArcGIS 10.2 is now supporting that, if I remember correctly. So they were super happy to see that they could just export to it, kind of. But they were on 10.1, so I didn't actually try it with them. But I showed them how to use OGR to OGR for it.
24:21
Well, I was just wondering how the PostGIS works because ArcGIS server published their data as a REST service. Does PostGIS publish their data as REST service or how the PostGIS works here? It's not only as a REST server. Basically, it's the same as putting any other spatial database behind your ArcGIS,
24:41
and then you select the kind of service you want to make out of it. Yeah. Yeah, so sometimes at larger companies, we have the problem where they're actually more comfortable if they get to invest money in the open source project. How would you suggest going about getting those doors open? I understand completely what you're talking about,
25:02
and that's especially on a private sector thing. I don't know of a good blanket answer for that. I think it depends on the open source project. Some of them have easy ways to do that, pay for support or pay for features, but I think that varies from product to product.
25:21
That was all. Thank you very much. Thank you. Thank you all. Thank you. Thank you.