Web and mobile enterprise applications
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 188 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/31701 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2014 | |
| Production Place | Portland, Oregon, United States of America |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G 2014 Portland68 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
Open sourceFront and back endsMappingRepresentational state transferBitSource codeUtility softwareMultiplicationCartesian coordinate systemEnterprise architectureSoftwareDebuggerTerm (mathematics)Device driverIterationBusiness modelGoogolMobile WebPoint cloudWorld Wide Web ConsortiumTelecommunicationNumberDemoscenePoint (geometry)Cross section (physics)Set (mathematics)Complex (psychology)Stack (abstract data type)Connectivity (graph theory)Product (business)PrototypeMoment (mathematics)SpacetimePopulation densityProjective planeVector spaceFocus (optics)Computer clusterStructural loadFamilyField (computer science)Multiplication signPresentation of a groupPhysical systemWeb serviceUniform resource locatorImplementationServer (computing)Condition numberDivisorCommitment schemeMathematicsClosed setCASE <Informatik>QuicksortProcess (computing)Combinational logicMixed realityCrash (computing)Slide ruleLevel (video gaming)Online helpGraph (mathematics)PressureCellular automatonData storage deviceAdditionExecution unitVolume (thermodynamics)Metropolitan area networkFreewareFunctional (mathematics)InferenceScaling (geometry)View (database)Dimensional analysisCommunications protocolSign (mathematics)Mobile appDifferent (Kate Ryan album)Type theoryScripting languageSpectrum (functional analysis)Software developerPrime idealGoodness of fitPredictabilityGoogle MapsWebsiteGraph (mathematics)Hybrid computerComputer animation
08:43
QuicksortInterface (computing)Operator (mathematics)NumberSystem callSpacetimePoint (geometry)Digital electronicsResultantSet (mathematics)CuboidQuery languageWindowTransformation (genetics)Cartesian coordinate systemField (computer science)Table (information)AreaParameter (computer programming)Type theoryTracing (software)Different (Kate Ryan album)SoftwareTerm (mathematics)Single-precision floating-point formatMultiplicationPhysical lawRight angleIntegrated development environmentLevel (video gaming)InformationFlow separation2 (number)Asynchronous Transfer ModeTwitterFront and back endsView (database)TouchscreenMobile appComputer animation
12:09
Cartesian coordinate systemDifferent (Kate Ryan album)Physical system2 (number)BitData managementSet (mathematics)Mobile appTouchscreenPresentation of a groupLatent heatComputer animation
12:58
Real-time operating systemLevel (video gaming)Pole (complex analysis)Greatest elementGame controllerPoint (geometry)Operator (mathematics)Multiplication signMobile appTouchscreenMotion captureField (computer science)Cartesian coordinate systemPointer (computer programming)Transformation (genetics)MappingComputer animation
14:10
Computing platformGeometryMobile appWeb applicationTerm (mathematics)Set (mathematics)Different (Kate Ryan album)SynchronizationMultiplication signWindowBackupDatabaseRevision controlDependent and independent variablesSoftwareStack (abstract data type)FrequencyBusiness modelDigital photographyField (computer science)DemoscenePoint (geometry)QuicksortAndroid (robot)Operator (mathematics)Replication (computing)Cache (computing)Demo (music)ArchitectureMachine codeServer (computing)LaptopBitWeb serviceLevel (video gaming)Open sourceSocial classSmoothingConnected spaceMathematicsFile formatUtility softwareReading (process)World Wide Web ConsortiumCharacteristic polynomialAttribute grammaroutputMultiplicationKey (cryptography)Java appletWebsiteScripting languageType theoryTesselationComputer configurationData storage deviceCapability Maturity ModelState of matterVideoconferencingPredicate (grammar)Cartesian coordinate systemSequelBit rateDirected graphCross-platformCode refactoringComputer file
18:46
Level (video gaming)Multiplication signZoom lensConnected spaceFunctional (mathematics)SoftwareWindowLocal ringBitGoogle Street ViewNetwork topologySampling (statistics)Set (mathematics)Business modelQuicksortWordComputer animation
20:20
Query languageBitTerm (mathematics)Product (business)PrototypeCASE <Informatik>Traffic reportingComputing platformSubsetMachine codeDifferent (Kate Ryan album)Mobile appDatabaseGame controllerMultiplication signConnected spaceData storage deviceComplex (psychology)Set (mathematics)VideoconferencingType theoryComputer fileDigital electronicsDemo (music)SynchronizationSoftware testingPhysical systemCartesian coordinate systemWorld Wide Web ConsortiumProcess (computing)Volume (thermodynamics)Enterprise architectureMobile WebCache (computing)Android (robot)Condition numberConstraint (mathematics)Moment (mathematics)Functional (mathematics)SequelMultiplicationAreaElectronic mailing listCausalityStandard deviationOverhead (computing)Sound effectBeta functionLecture/Conference
24:14
Medical imagingAreaCache (computing)Fluid staticsScripting languageComputer fileCombinational logicMachine codeDifferent (Kate Ryan album)QuicksortJava appletTesselationSubject indexingSingle-precision floating-point formatBitElectronic mailing listAttribute grammar
25:01
Machine codeLimit (category theory)Game controllerLinker (computing)QuicksortWebsiteWeb serviceBitBounded variationDifferent (Kate Ryan album)Cartesian coordinate systemMechanism designWeb browserLocal ringCache (computing)ConsistencyIntegrated development environmentType theoryVolume (thermodynamics)Multiplication signMultiplicationAreaSingle-precision floating-point formatTesselationField (computer science)Slide ruleComputer configurationCASE <Informatik>NumberData managementComputer fileSelectivity (electronic)Connected spaceSoftwarePoint (geometry)outputData storage deviceWorld Wide Web ConsortiumOperator (mathematics)Square numberSemiconductor memoryDigital electronicsProjective planeAttribute grammarDialectWordDatabaseLevel (video gaming)Graphical user interfaceSequelWeightLaptopVector spaceAndroid (robot)Goodness of fitLatent heatWindowRight anglePerturbation theoryAbstractionTheorySystem callLibrary (computing)Computer programmingMobile app2 (number)NP-hardFlow separation
29:53
Level (video gaming)Field (computer science)Cartesian coordinate systemOffice suiteComputer configurationWorld Wide Web ConsortiumEmailCASE <Informatik>Scaling (geometry)outputLocal ringPlastikkarteAreaBitServer (computing)SynchronizationDatabaseData storage deviceQuicksortSet (mathematics)Computer fileWeb browserTesselationInformation securityMathematicsSoftwareLimit (category theory)Port scannerReal numberDegree (graph theory)Utility softwareSelectivity (electronic)Uniform resource locatorExterior algebraBasis <Mathematik>Sound effectMappingWechselseitige InformationMobile appComputing platformCache (computing)Connected spacePeripheralDifferent (Kate Ryan album)Open sourceRow (database)Client (computing)Enterprise architectureData modelPropagatorOpen setScripting languageUniverse (mathematics)Decision theoryPoint (geometry)System callInstance (computer science)Right angleVolume (thermodynamics)Touch typingCharacteristic polynomialEvent horizonLine (geometry)Structural loadOntologyPrisoner's dilemmaComputer fontMultiplication signProof theoryExistenceDigital electronicsComputer animationLecture/Conference
Transcript: English(auto-generated)
00:02
OK, thanks for coming, everyone. I'm Peter Batty with Ubersense. And I'm going to talk a bit about what we're doing with sort of large-scale web and mobile enterprise applications. I'll just, to begin with, talk a little bit about the background on our customers and the software stack we're using,
00:22
and just very briefly touch on business models that came up in a couple of other presentations. But the two big bits that I'm talking about really are simplicity, which is a theme we've heard about in a few other presentations, and then field applications online and in particular offline. So those last two are the kind of main chunks that I'll talk about.
00:41
And I think we'll risk a live demo, which is always fun with conference Wi-Fi. But so our customers really are large utilities and telecom customers. In general, the bigger end of the scale. Basically, a lot of folks in our company used to work at a company called Small World.
01:01
And I'll come back to that in a moment that was in that space. These customers tend to have very dense vector maps, these kind of complex network maps, and large data volumes. They have up to five, six, seven million customers. They handle, in some cases, quite complex telecom networks,
01:22
things like this where you have cross-sections of networks, a lot of fairly complicated things to handle, even going down to inside plants in some of the telecom applications where you're modeling things like this all within the GIS. And so the three big traditional players in that space are these three, GE Small World, Intergraph,
01:42
and Esri, all traditional closed source GIS products. Sophisticated, but kind of complex, too. In previous lives, I was CTO at Small World and Intergraph, so I know this space well. I've worked around this space a lot.
02:01
So say, you know, incumbents, large, complex, sophisticated systems. But then on the other side, on the other end of the spectrum, you have location being pervasive and simple in all of these consumer applications now. So our big focus the last few years here has been to say, what can we do to take all of these ideas and technologies
02:21
from the consumer space and apply them to these, you know, probably some of the most complex enterprise applications, but really take that data and make it available much more simply to lots of users. So yeah, just an example of keeping it simple there. So I'll just briefly mention our software stack
02:42
and how we got to where we are. So when we started three, four years ago, we did our first prototype. I used a product called Arc2Earth, which as the name suggests, was focused on storing Esri data in the cloud and integrating with Google. That ran on Google App Engine, and we used Google Maps on the front end.
03:03
And basically, you know, with our customer base, they're really very resistant to the cloud. These large utilities and telecom customers are, you know, very conservative IT-wise, and they just said, we love the concept of what you're doing, but we don't want it to be in the cloud. And so we transitioned our back end
03:20
to PostGIS and Mapfish. We're not really using a lot of the features of Mapfish. It's quite a rich product, but we pretty much use that just for a REST API in front of PostGIS. So, you know, there's possibility we might use some other things in future there, but it's worked well for us up to this point. So we developed most of our application on Google Maps.
03:42
And then next iteration came when we had a lot of customers wanting to do offline things, but Google is basically focused on online, their terms of use specifically prohibit you using either the JavaScript or the data offline. And so that was a primary driver for us to switch to Leaflet.
04:00
So basically our whole software stack, enabling software stack is open source right now. And we still use Google data a lot, then we use Google Street View. We like that, but because of the offline apps, we basically moved over to Leaflet. And so as you can see on the left hand side, we can use multiple data sources,
04:20
whether it's Google, OpenStreetMap, whatever, as the background maps. So that's just a little background of how we kind of got to where we got. And I thought I'd just mention, a quick show of hands actually. I mean, who predominantly uses open source for that geospatial stuff here? And who uses a significant bit of non-open source?
04:41
Okay, so a fair mix. So I think it's just worth saying this. I mean, to many of you, this may be obvious or preaching to the converted, but I don't have any sort of innate predisposition to open source or closed source. I mean, increasingly, the more I work with open source, the more good things I like about it. But if a closed source system is gonna do the job better for me,
05:01
then I'm gonna choose that other factors permitting. But so we didn't come into this saying, hey, we want to use open source, but just through a combination of the factors on this slide, we've gradually over time moved more and more towards an open source stack because it meets these needs best.
05:20
And I went towards this in great detail, but clearly you need the right functionality, otherwise none of the rest matters. It's interesting how many people downplay the cost, but to me, it's still important that it's free. Hey, I mean, don't understate that. I mean, the software licenses are free. Again, as other people say, doesn't mean the whole implementation is free.
05:41
But for quite a lot of reasons, especially when you're working with large numbers of customers out there, you want to scale up your servers and you don't have to get additional license costs, that kind of thing. That's important. Support, our experience in the open source world has been great. By and large, we haven't needed it. The one time we had, we threw a load of new GeoServer things into production,
06:02
helping recover from Hurricane Sandy. We were getting crashes a few times a day and we couldn't figure out why. And we got an overnight fix from Andrea, I am in Italy, which is something you'd be really hard pressed to get from most of the big geospatial vendors. And obviously your mileage may vary,
06:21
but that was the one time we needed a critical fix and we got it super quickly. Terms is another important thing. Like I mentioned, the big driver was being able to work offline and we were able to do that with Leaflet and OpenStreetMap. We couldn't do that with Google. So in a lot of cases, that there's more flexibility there. And predictability, you know,
06:40
the thing is, with a lot of commercial vendors, you don't know if they're gonna discontinue a product or change terms, change pricing, that kind of thing. With open source, you can sort of have a long-term commitment where, you know, even if something kind of happens with people moving off the project or whatever, worst case you've got the source code. But in general, if it's a dynamic community, you know, there's much more guarantee
07:01
that it's gonna be there and on the same terms and conditions tomorrow. And then I'll just very briefly mention business models. So this came up in a couple of other presentations. So I think if you're working with open source, you know, there's multiple ways to make money and we all need to feed the family somehow. You know, there's companies like Boundless
07:21
who sell support and services. The software is free. CartoDB and Mapbox sell hosting. But then there's a number of companies where I say, if you like, they have hybrid, closed, and open source. So they're actually selling quite a lot of products that are closed source, but they're still quite engaged with the open source community. You know, Google is one good example. Primary products are closed source,
07:42
but they really do a lot of work in the open source community. They sponsor a lot of open source initiatives. They've released a lot of open source projects. Safe Software, I was talking to earlier, they're similar on a much smaller scale, but you know, they use quite a bit of open source software and contribute to that. And we're basically the same. So we sell the products that we're showing you here
08:03
to the large utilities and so on. But you know, we're quite engaged with the open source world in multiple ways and giving back in different ways. And I'm hoping that over time, we might possibly even open source some of the components of what we're doing. But so anyway, so let's get onto the main topics that was in many ways, preamble and miscellaneous things.
08:22
So simplicity is one of the first things I wanted to talk about. So this is just say this kind of general aim of what we talked about. And so I'll just give you a quick demonstration of what we're doing and try and talk through some of the things we're trying to do to simplify, say what are really some of the most complex sets of data and applications behind the scenes.
08:42
So this is our main app, as I mentioned, using Leaflet. We're using PostGIS on the back end. So we come in here, oh, that's the wrong place. So another thing I'll show you here is, if I can see the screen here, it's got a lot smaller.
09:03
So I'll show you the search bar more in a moment, but we can click on anything on the map. One thing I'm big on is not having these separate info modes to click and that kind of thing. We can look at Google Street View. One thing that's kind of nice, a lot of people aren't aware of is in the Street View API
09:20
you can embed things in there. So we can look at poll numbers, we can look down the street, see the next poll numbers. And there's a lot of applications for that and the sort of things we're doing. Often it can save people going out in a visit to the field. But you can see all of this looks very much like Google Maps or OpenStreetMap, the kind of things that people will have seen elsewhere.
09:43
We do a lot with this search window here, sort of single box search. Again, trying to simplify a lot of typical GISs have queries where you have to specify tables and field names. Here you just type a poll number or a valve number or whatever you're interested in. You can save bookmarks here.
10:01
And let's see if there's an area here. And we do a lot of queries through this search box too. So again, trying to, if I just start typing poll, I can say query polls in window, you get all of those back. So trying to have this very sort of intuitive interface here.
10:21
And then one last thing I'll show you is trying to simplify more complex operations from the GIS. One thing that you have in this kind of space a lot is network tracing operations, where you're tracing through the network and there's all kinds of different parameters to trace upstream or downstream and look at things.
10:42
And we try to really simplify that to where the sort of users that we're targeting don't know about GIS and they don't know about traces, but they do know the concept of the poll upstream or the device upstream from where you are now. So we've just sort of pre-calculated a lot of these things. So if we click here, we can say show me the next upstream device.
11:01
This is a transformer I've clicked on. And it shows me that switch is the next upstream device. And then the circuit is the set of all the devices that's fed from a single point. If I click on show me the circuit, then you can see it very quickly highlights that circuit and you can jump to it. And those kind of things typically take multiple seconds back in the GIS at the backend.
11:20
So again, we've really tried to think about a, pre-calculating results from more complex operations, but that these large users are, or, you know, these large numbers of users are gonna use and then really presenting it in a way that makes sense to the user, not in terms of GIS operations kind of thing. And then finally, the thing, I saw a tweet from Javier from CartoDB yesterday,
11:44
and he said something like the future GIS is not one application with 100 buttons, it's 100 applications with one button. And that's a philosophy that we subscribe to. There may be not one button, but maybe three buttons or five buttons. And so this is something we build into our app is the concept of multiple applications.
12:02
So here, you know, if I click on the standard application here, it comes in with a bunch of different layers, electric network and whatnot here. If I go back to that main screen, I click on vegetation management instead, it's a different set of layers that are relevant to that particular application. And it's got a specific toolbar down there
12:22
for that particular application. And again, that's something that we really try to do is to say, let's have a bunch of different focused applications for different users. So it's important to build that into the system to make it just easy to deploy those things. So those are a few ideas I wanted to share just on keeping apps simple. Again, it's a theme like we heard from Vladimir and so on,
12:40
but I'm very much a believer in that. So let's switch back to the presentation. I'll talk a bit more about the second topic offline. I guess I skipped a couple of things here. I'll just quickly run through some screenshots showing an example of the kind of applications we want to run offline with these customers or in the field.
13:01
So damage assessment is a key one. After a storm, all these poles are knocked down and whatnot, they want to send people out and evaluate the damage and be able to push that back in real time as much as they can to the operations center. So here's the operations center. This is a screen on a desktop.
13:20
Nothing started yet, there's no damage shown. We switch to a guy in the field and he has a pretty simple application where he can click on a point, enter information, saying there's a pole down here and create a damage point. And then back in the control center, assuming the guy's online, you see the little eye where that damage point is and then you begin with a thematic map
13:41
saying we have some damage in this county here. And so that's kind of maintained as you go along. And so then if we go back to the field, the little thing down at the bottom is saying he's offline now. Especially after a storm, but also just because these guys work in rural areas, they will be offline quite a lot of the time. So the aim for the app is you need to be able
14:01
to seamlessly switch between online and offline, capture data offline, but as soon as you connect it again, push it back. So he can go ahead, the guy in the field, create a couple more points. So now we've got two points out in the field. He's offline. So if we go back, there's still just that one point back in the operations center.
14:20
He reconnects now and then it automatically syncs behind the scenes and now you've got three of those points back there. So that's just to give you a kind of flavor for what we're trying to do. And then another requirement is capturing photos in the field and pushing those back to, again, the same general wish that they want to be able to store those offline.
14:41
And so overall, the requirements for the offline app, you know, really very strong desire for it to be cross-platform, Android, iOS, Windows 7 and 8, and then also web. So we really want our existing web applications to be able to run there. So we'll talk about the different approaches. We do need the ability to sync large data sets
15:01
for our app. I mean, potentially gigabytes of data. A lot of these utilities would like to sync either all of their database or a large region because they might, you know, especially in these damage or emergency response situations, they might be offline for quite a while. In other situations, you might be able to use smaller data sets like a single circuit.
15:22
And that enables a different technology approach, HTML5, which I'll talk about shortly. In terms of the characteristics of that sync, there tend to be two different types of data. There's your sort of bulk GIS data, which for these customers doesn't change very rapidly, but you know, it does change daily.
15:41
But so there you have this bulk read of multiple gigabytes typically, and then you really want to do a nightly incremental update, but nightly is good enough for that. But then you have these more time critical data sets like the damage assessment data where you want to sync as soon as you're connected. And there you really need to connect and disconnect and sync transparently.
16:01
So there's basically three different architectures we tried to do this. And we'll talk about each of these three. So I won't dwell on it here, but basically two using replication, and where you really store a lot of data offline, the other one using HTML5 and more dynamic caching. So the first one is really just running a replica
16:23
on a Windows laptop. This allows us to use exactly the same stack as we had on the server with PostGIS, Mapfish and everything. So in many ways, this was the sort of simplest option, but it only runs on Windows or Linux or Mac. But this won't port to Android or iOS because of the software stack. You do need sort of custom replication for this.
16:43
At least I haven't found any replication for PostgreSQL or PostGIS that sort of handles this model where you're disconnected for long periods of time, and then you re-sync. There are replication solutions, but they're more for kind of backup or failover kind of things where they really assume it's connected all the time.
17:02
So this one is relatively simple to implement, although there is quite a lot of detail in doing a custom replication. The cons are it's laptop-only and a fairly heavyweight software stack. But that works. So the second approach we used was PhoneGap,
17:20
a PhoneGap replica. And so the idea here was, we've had a few other people talking about PhoneGap or Cordova, basically the same thing, Cordova's the open source version. And so for anyone who doesn't know, basically what this does is it takes your web application, HTML and JavaScript, and it compiles it to a local app.
17:41
But it is up to you basically to take the data and have that stored locally. So to do this, we basically replaced PostGIS with SQLite, which runs on all of the platforms we needed, both for geometry attributes and auto, so to store the tiles, basically in an MBTiles-type format. We also had to rewrite a little bit
18:01
of our server-side code, because that was written in Mapfish and Python. So for the key services that we needed, we kind of rewrote those in JavaScript to wrap the database. And then kind of refactored the code just so those differences were in common classes. So the great majority of the code stayed the same.
18:20
And that's what it looks like just on a chart. I won't dwell on that now. Let's see how we are. So I think I will just, we've just about got time and we got a break before. So I'll give you a quick two-minute demo just to give you a feel, because there was quite a bit of discussion in earlier sessions of how well does PhoneGap work, is the map smooth enough, and that kind of thing. So I'll just let this video play.
18:41
It's about two minutes. And we come in kind of halfway through when I'm just going offline. I'd shown a few online. Go offline. So if I switch to airplane mode, we lose our network connection. And what you'll see now if I zoom out a little bit is, oh, we actually have some data cache there.
19:01
But in general, I'm starting to see now that I have no background map, because I have no connection to Google. Here it's saying, sorry, we have no imagery here. And we can set this up to flip automatically, but basically I can say I'd like to switch to have a local OpenStreetMap backdrop now.
19:20
So now you can see this is a locally stored base map. We've still got the same capability to pan and zoom. And this particular base map just covers certain zoom levels. Let's zoom in again. This time if I select the same poll, you see there's no street view here.
19:41
It knows it's offline, so it just doesn't show that. But we have other capabilities that we can show, like if we want to see polls in window, for example. We can run that, and it shows all the polls in the window here.
20:03
Click to go to a specific one and that kind of thing. So regular kind of functionality that you have. I can search for poll number, for example. Go there. So again, all the same functionality that we have when we're regularly online.
20:21
So I think that shows you the principle. I go through a couple more things. But basically, we're just porting everything over to SQLite. I mean, in effect, you do get all the same functionality, but just, except for things where you obviously need a connection like Google Street View and so on. So in general, we found that that worked pretty well. But there was quite a bit of detail
20:41
and fiddling around. Again, there was some discussion earlier of people's experience with PhoneGap, but overall ours has been pretty good. You do hit odd little tricky things where you have to spend time just figuring out what caused that difference between the two. But we've been working on it for a little while now and it seems pretty robust and works pretty well for us.
21:02
And basically, the benefit of having one common code base for web and mobile across the different things, I think makes us lean pretty strongly towards that versus going to developing native apps for the different platforms. Obviously, it might depend on your requirements a bit, but for us, that's been the case. One caution with iOS, we have increasingly,
21:22
while I'm a bit of an Apple fanboy in general, I have got quite fed up with Apple in terms of these enterprise applications, that their terms and conditions can be quite a pain and they've tightened them up quite a lot. You know, one basic thing is that Apple has to review all code, including fixes and whatnot, which can be a multi-week process.
21:43
And it used to be that you could deploy testing systems with this system called TestFlight, but they've clamped down on that in the last couple of months. So even quote, quote, beta things that you're deploying through TestFlight have to be approved by Apple. And so there's quite a bit of overhead there. And then our apps tend to be these big complex things
22:02
that are customized to each customer. And there's, Apple has a couple of different ways of distributing those so that apps don't go through the app store, they're specific to a given customer. And I won't get sidetracked on that, but just by way of warning, just that there is quite a lot of complexity here and we've hit quite a lot of issues. So we're encouraging people pretty strongly
22:21
to look at Android for these kind of enterprise deployments at the moment. Hopefully that will improve, but that was our experience. And then the final thing I wanted to talk about but was the experience with HTML5 offline. So there's constraints to what you can store with HTML5 offline in terms of volume of data,
22:42
but it does work quite well with smaller amounts of data. So in particular for that damage assessment application I showed you, quite often what they'll do is walk one circuit at a time, which is a relatively small area, they'll give that to one person. And so it's quite reasonable to download that using HTML5.
23:01
And this is much more dynamic, you don't have to install apps, you don't have to do a big data sync initially, that kind of thing. I think in the interest of time, I'll skip the demo on this one. If anyone's particularly interested, I can show the video afterwards, but I'll just explain what we did. But essentially it's very similar functionality, but it's a bit more limited because you don't have a full database with HTML5,
23:24
at least with what we've done now. So it is more of a subset of the functionality, but you can basically pan, zoom, select, you can create these damage reports like we saw before and push them back. But we haven't implemented anything like spatial query or anything like that. And this is still prototype stuff right now.
23:40
I mean, the other stuff that we've done is pretty solid product. We've been prototyping this stuff. So I just wanted to explain a couple of aspects of HTML5 for people who are not familiar with it. There's kind of two separate things when people talk about offline storage. Well, two with variants of them, but one is what is called app cache
24:01
or cache manifest files. And this basically lets you list a set of files that get cached when you're offline. And then there's offline storage and there's various types of that. I'll talk about where you can more explicitly store things under programmatic control. And this is what a cache manifest file looks like. I have to put a little bit of code up here at some point.
24:21
But basically the top files are sort of static HTML, JavaScript, CSS files, so that's relatively static. And then beneath that, we've got a bunch of image PNG files listed. And then with what we're doing, we tried a couple of different ways of getting attribute data down. One was doing the tile JSON files, which gives you sort of a simple spatial indexing
24:42
if you click somewhere, you can figure out which tile would it be and then look there for that. Or just a single geoJSON layer if it's a relatively small area. But anyway, so you sort of can dynamically create this cache manifest file for a certain area. So it's a combination of the static stuff up front and then the dynamic tiles for the area that you're in.
25:02
Just a couple of other basics on, and so on the offline storage side, which is separate from that cache, there's multiple different types of offline storage. And there's quite a bit of inconsistency between different browsers. I won't get into the detail here. There's also quite a lot of variation in the offline storage limits.
25:23
I've got links on this, but these are all from HTML5 Rocks, which is a .com, which is a really good site for this kind of thing. So anyway, those limits vary quite a lot. But a library that we found that is really good is called Large Local Storage. And that abstracts all of these differences
25:40
and in theory gives you unlimited storage up to any hard limits on the device. And it uses different underlying storage mechanisms. So we've used that and definitely recommend this as a mechanism for working across browsers or for using offline storage. So anyway, back to the specifics of the app.
26:02
We found that caching individual tiles using the app cache, it works reasonably well for fairly small areas. So like one electric circuit, which is probably less than a square mile or something. Our example, we had 800 tiles and it maybe take one to two minutes to download that on a reasonable network connection. So that was kind of manageable.
26:21
But there's a couple of drawbacks. I mean, one is that app cache is all or nothing. If it fails at any point, then the whole cache gets rolled back. You have to download every single file. So that's not very good as you get into larger data volumes. Also just, maybe this is on a separate slide here,
26:41
but basically it's okay for smaller areas, but just copying lots of small tiles is intrinsically slow, right? Whether it's on a web environment or anywhere else. And so when you get to really large data volumes, doing a tile at a time doesn't work very well. And I'll come back to that in a second. The other thing is that there's no real ability to manage or delete those caches programmatically.
27:02
That you can't get at that from your application code. So the user has to know more technical stuff if they want to clear caches out and that kind of thing. Let me down there. So anyway, and so this mechanism does work fairly well. You can also do things to cache individual tiles in local storage using more programmatic control.
27:23
That means writing a little bit more code, but it does give you more control over this thing that it's no longer all or nothing and you can delete them out more and whatnot. So for sort of a serious effort, that's probably more worthwhile. Yeah, so I guess it's what I was saying. For individual things, for individual tiles,
27:40
it's very slow to copy them. But so we have done some experimentation recently in downloading MBTiles files. And there's a library called SQL.js, which actually I was pleasantly surprised how well it worked. And so we were able to download an MBTiles file, which is like massively faster than downloading the individual files,
28:01
and then unpick it using SQL.js. And that actually worked pretty well in Chrome on a laptop or a high-end Android for a file that was 275 megabytes, which is a pretty good size area. It was tiles down to a fairly deep level for a city. We had to overcome a few memory issues and flakiness,
28:21
but we got it working reasonably well. But unfortunately, our main target for that particular project was we wanted to run it on iOS and we just kind of naively assumed if it worked on Chrome, it would translate. But it turns out that Chrome on iOS is built on WebKit and has the same 50-megabyte limit as all the other browsers on iOS.
28:40
So there is still quite a hard size limit for this on iOS but for Android, for a high-spec tablet, and for Windows or other things with Chrome, this does look quite promising. So this is work that we've done quite recently and we haven't fully followed through on that. But I think using MBTiles or some other consolidated file
29:00
is something that you really need for larger areas just because it's not very practical to copy down the individual tiles. And then finally, I think I mentioned this briefly, but so for attribute data, I guess you could do more with the local database and so on and construct something more complex. I mean, we're trying to keep this fairly lightweight. So we've used two main approaches. One is GeoJSON tiles, like I mentioned,
29:22
where you tile them up and you can do that basic select operation. You can just figure out which tile that we click in and then you have a manageable number of features there to search through and make sure you can find what you selected quickly. Or the other thing is, say, for relatively small areas, just download some GeoJSON for the whole area and Leaflet handles that pretty well,
29:40
the selection and everything. And then lastly, just a couple of other options I thought I'd mention in case you don't know them. One thing is a thing called FieldPapers, which has come out of OpenStreetMap, used to be WalkingPapers. But this is kind of interesting for certain applications, you can just print out a paper map, you can sketch on it,
30:01
take a picture with your cell phone and email it back in and then it orients that scan map, behind your web map back in the office. So it's a kind of low tech solution, but it can be useful for capturing the field. So that's just one interesting other option for field applications. And then one other little thing
30:20
we experimented with recently is either maps on a USB stick or this thing called an Airstash I just got, which is like a USB stick, but it has a Wi-Fi thing built in. So this is actually a solution that could overcome that limit on iOS that if you're just storing tiles and JSON, you can't run any server software on this, you're purely accessing files,
30:40
but you can do basic tiled maps and this simple JSON selection using that. And you can stick an SD card in that, I got 128 gig SD card, so, you know, no real issues on size, but the particular customer we were talking to didn't want a separate device, they wanted it all on the phone. So they didn't like that, but I thought it was kind of another cool option to explore.
31:02
So anyway, quick summary on the offline. I mean, it will be great when we have universal wireless coverage, it will simplify all these problems obviously. Today offline is still a bit harder than you might hope. I think large scale robust sync for enterprises isn't rocket science, but there's a lot of detailed work to do well
31:21
and there isn't really anything existing in the open source world that I know of that does that, quite what we need there. HTML5 caching does have a lot of promise and I think it's workable in some scenarios, especially for quite small areas, but it's still not quite fully baked, you know, it's a little bit flaky, it's not well documented, all the differences between platforms and whatnot.
31:42
So hopefully that will really improve in the next little while here. And then there are these alternative options like field papers or external storage or whatever that may work in some cases. So anyway, that's it, quick run through what we've done on offline. I don't know if there's any, and I was touch over probably,
32:00
any quick questions? One at the back there. Yeah. Oh, sorry, well, I guess we've got a mic here, so I'll take this one first and then. I'm wondering when you have somebody like an electrician or a field engineer going out to fix a pole for instance, right? And so maybe they're offline because it's in a disaster.
32:20
How do you figure out if you're gonna give them scanned map tiles for local or imagery? And when you do make that decision, how much, how big of an area? Yeah, so that's a good question and it does depend very much on the work, the workflow, you know, you have to understand what they're gonna do. So in this damage assessment scenario,
32:41
usually they go and walk a whole circuit, which is a predefined area. So we'll give them all of that. And then we kind of agree with the client, you know, do you need imagery or do you just need an open street map, base map or whatever? So that in our apps, that tends to be all kind of decided ahead of time, pretty much that we look at the specific workflow.
33:01
So I don't know if that kind of answers the question, but it does depend on the scenario. You know, another common one with our customers is they, when they go into a manhole, they lose wireless connectivity. So that's something that you can predict. So you could have a button that says, hey, I'm about to go into a manhole and then you presumably just need a pretty small bit, but they still want to be able to look at the records
33:21
and that kind of thing, so. So earlier in your slides, you mentioned replicating from post just on a server down to spatial light on the devices. So is that something you guys got working well or, and if so, what were your experiences with that?
33:41
Yeah, we do have it working pretty well now. And I think, did we use OGR to OGR, Mike, because there's sort of basis for that. So, yeah. From the, so use. Yeah, yeah, sorry. So yeah, we use OGR to OGR or OGR to OGR
34:02
is the basic thing, but then. Or use that on the devices or on the server? On the server. So it tends to be that the way it works is, you know, we package up, I mean, we typically have several hundred to several thousand offline users. So it's usually packaging up a database once, which might be the whole thing or might be a subset.
34:23
And so we have it working pretty well, but it has been quite a lot of work to get there. You know, it's like this thing where there's just lots of little details and what happens if you have a data model change and there's a whole bunch of things when you're doing large scale deployments that get kind of to be a pain and they're just, let's say they're not rocket science,
34:40
but it's just a lot of little details. So, you know, it has been a fairly significant effort to get it going. So when you say you use Ogr2Ogr, do you create a, like a spatialite database of changes that you send to the devices or are you sending GeoJSON or? Yeah, so we do this initial load and then we do the nightly changes, which will package up.
35:02
And again, we do one package of nightly changes and propagate that to all the things. I think, are those changes GeoJSON, Mike? Do you know? And then we zip them up basically and download them and then run a little script on the client that kind of unpacks them basically.
35:26
Okay. Thank you.
35:44
What are the advantages to PhoneGap versus HTML5? Really the biggest advantage to PhoneGap is having this larger storage capability. So there we can have a native SQLite database and we can store multiple gigabytes. That's the biggest single thing that there are,
36:01
you know, you also have better access to native things on the phone, which we don't need for most apps, but we do have one app where we interface to like Bluetooth gas detectors. And so for that one, we're gonna need the native app. The advantage of the HTML5 is it's on the whole
36:21
a lot easier to deploy. Like a scenario with this damage assessment is you get what they call mutual aid crews who come in from a different utility. And if you can just send them a link and like in two minutes they have a working app, they can unplug and go, that's huge. And obviously shipping updates and all that kind of thing is a lot simpler, but the biggest limitation with the HTML5 is, well the biggest single limitation
36:41
is the data volume. And then say just still it's a bit of flakiness and you know, so it's definitely promising. I think especially as you get better and better wireless coverage, I think more and more cases will be covered by HTML5. So I expect over the next couple of years we'll do a lot more with HTML5, but we still kind of need the base capability
37:02
of syncing a large data set offline. Just a last question. Peter, I was wondering if you discovered a way to take a cache manifest and a package of files, sort of package them together and deploy them without having to put them up on a web server first.
37:23
Is there a way that you can put those on the device without having to have those on the web server first? Well, kind of, I mean we sort of looked a little bit at this idea of maps on a stick, a USB stick or an SD card, and in a lot of cases
37:42
with the stuff we were doing with HTML5, you know in effect you end up with an app that is running offline where there is no server. And so to some degree with fairly minor changes you could often package that same set of HTML files, just put them on an SD card or whatever and then just run from that locally,
38:01
which has a few slightly different characteristics, but broadly speaking will work. So like this wireless device that we had, we used more or less the same set of files that we'd cached, all the tiles and everything, but you just had a simple app that was going against these files stored on the device. But I don't know if that, does that make sense?
38:23
File URLs then instead of HTTP? That varied slightly. I think in some cases file URLs work, in other cases the security on the browser might not let you do some things with that. With this external device with the WiFi I think we could actually access them as HTTP files,
38:42
that air stash device, so that seemed to work, still serving them through HTTP, though we couldn't run any extra software on there, we could just serve the files. Okay, all right, thank you. Yeah, thanks, and if anyone has other questions I'll hang around for a bit. I also have the iPad offline app here if you wanna just have a quick play with it and see how the PhoneGut thing works.
39:02
Yeah, thanks.
Recommendations
Series of 3 media