This is a talk on GIS in Node.js. Hi, I'm Calvin. Follow me on Twitter if you care about obscure JavaScript and mapping things that I talk about. My website, and then you can follow along here, which I had up about 30 seconds ago, so you saw that.

So that was supposed to be a crossed out line through there, but imagine that. So if I go too fast, don't forget, don't hesitate to raise your hand or just yell at me. This is going to be somewhat of a closed line talk.

So it's sort of a very sort of broad surface area of GIS C stuff in Node. And we're going to cover a lot of them in a little bit of depth, but not necessarily in detail. So if something makes no sense or I start babbling

about incoherently, raise a hand or just yell at me. So first of all, what is Node? People often talk about Node in sort of comparison to like Django or something. So in Python terms, it's not Django.

Django is other stuff. It's a templating language. It's sort of a whole stack. It's not Flask, which is the other thing I know about Python. It's equivalent to Python. Like Node and Python are the equivalent things. It's a language. It's a runtime VM, interpreter thing. It's a standard library.

Though unlike Python, Node.js is very much batteries are not included. You can bring your own batteries, thank you very much. You're an adult. Though the exception is a package manager unlike Python, which has everything including kitchen sink, but not a package manager. Node does come with a package manager,

which is one of its very, I don't, it's one of the like the best things about Node. It's like you use Python, you use Node, you're like hell no. Pip. Once you use NPM. Yeah, but why? Why would you use Node? It's concurrent, which is one of those buzzwords

that people use for a lot of things, which if you ever use Vladimir's bullshit library, it will probably change to bullshit and you will not have any difference in the meaning of a sentence that you use concurrent. That means, mainly it means callbacks. So if you've ever told somebody

I'm not gonna wait on hold, just call me back and you've got the answer. That's how callbacks work. It's that if you're gonna read a file or download something from a web browser, instead of just stopping the program, waiting for it to be done and giving it to you, you just give it a function and then whenever it's done, it gives that back to you and you can continue going on your merry way with the rest of your program

while you're waiting for that thing to be downloaded. So that means that Node.js worked really good for IO bound stuff. It sort of, even though it's single threaded, you can do stuff like servers, stuff that have to read a bunch of files or write to files. It ends up being a lot faster than you would expect

for just a single thread sort of operation because you're not sort of busy waiting like you would be in a lot of other languages. So as I said, servers, HTTP requests, reading files. So one of the things is where am I gonna store my data

because a big thing in GIS is I got a crap load of data that I can't hold in memory. What am I gonna do with it? So Postgres. So there's some great Node adapters for Postgres. There's pg.js is a pretty simple one and then connects. It's like a query builder slash uses pg.js

and for querying, making complex queries, and then getting the data back. We use them both in production. They work fantastically. Postgres is really the best SQL database especially for geo stuff. And also when it comes to querying, it's the best JSON database.

It's much faster for querying JSON documents than JSON-specific databases which is kind of embarrassing for Mongo. That being said, we've got some cache base and cloud end people here. Your stuff is, I should have put in the open source querying stuff at the moment, open source.

Yes. The other, if you were at my talk yesterday, you heard some of this, but level up is also great back end for sort of like a, it fills a similar use case to sq-lite but it's sort of a much more basic key value store.

So maybe in like a Berkeley DB or a simple DB for Python, if you wanna like to store some stuff and you don't wanna deal with the file system, it's a pretty good one for that. But since it has ordered keys, you can do some pretty sort of neat stuff involving sort of key prefixes

and I did a talk yesterday about geospatial databases in level up. You can do some pretty neat stuff there. But the other cool thing about that is it has pluggable back ends. So you can do a memory using the browser, all of that stuff, React, all those.

And there's a big ecosystem and that's one of the things that you'll get in Node is that you have a lot of people who have already done, who have written something for this, but it's not a big framework or anything. It's just a small little module that does a small little thing that solves your one problem.

And that ends up being that you can then, oh, this does a quick little like tries for this or does, lets me put in some weird data type in. Somebody may have already written that small little module

and then published it for you to use. So PouchDB is another good back end in Node. Again, just on database, but this one syncs. Master, master replication, which is good for offline stuff and again, multiple back ends because it uses level DB and all the stuff we just talked about

and just using on Node, you can use the browser because it also has, and then it has the browser side storage, which you never want to deal with directly ever because they're horrible, horrible APIs, horrible. And you can talk to Cloud and you can run on the server.

So what are the sort of the cool things that people have been doing in Node with GIS that make it very easy to do GIS stuff with it? First of all, it's GeoJSON. It's the worst spatial, geospatial format except for all the others,

or at least for interchange, for giving people data. Just think of every single headache you've ever had with a shapefile and you have different headaches with GeoJSON, but they're smaller ones. You can do some really cool stuff with GeoJSON.

A lot of people have been doing stuff with GeoJSON in Node because it's how a lot of people are wanting to get their data. And that's from the original talk I gave, ignore the slide. I gave a preview of this talk about a month ago

and somebody, my opener was talking about GeoJSON. So TopoJSON is the other, is a really neat thing in Node. Mike Bostock, who hopefully all of you listened to with rapt attention earlier, did a TopoJSON, which is sort of lossy compression of GeoJSON with web maps in mind.

If you are making maps for the web, which is really the only way people actually want to consume a map, except governmental managers, TopoJSON is literally built for that. It is ignoring all the detail that nobody cares about

when you're in a map where the pixel is 10 miles. You don't need the sub-meter precision of floating point. No, it is exactly the size of the pixels, if not less. And TopoJSON is just built from the ground up for that

and it can make, I think, like a eight megabyte state of Massachusetts, all the cities and towns, put it through there and it was like 200K and you couldn't even notice the difference. So it's really helpful.

There's also a horrible Python port that I wrote and then never really, and then I got a new job with Applied Geographics and have never touched Python again, so if you want to maintain it, go for it. There's also a bit of a spec for that so that if you want to write,

I know somebody's written a Java port of TopoJSON, so favorite language that's not Node, go for it. Another really cool thing that you get in Node is streams, which are sort of built in for handling sort of HTTP connections, raw TCP sockets, but you can use them for things like,

but they're sort of just generic long pipelines of operations, so that you don't have to have it all in memory, so only a small part of a massive file in memory. So you can use these with GeoJSON because a lot of people are doing GeoJSON stuff, so you can use it to say take a giant file

that's like a couple gigs, put it in Postgres, leave it on overnight because there are streams for GME, Esri, PostGIS, and the guy who brought Promises to the web is doing a spec for getting these into your web browser,

so this is very useful once you sort of get your head around how it works. I've used it for geocoding. I had a couple hundred thousand addresses that I needed to geocode in a database and the Massachusetts GIS department,

they took about like 10 seconds to geocode to respond to the request, so you know, that's not something that you can easily do with just sort of iterating through them because you will immediately sort of run out of memory and crash your program.

So I'd set that up with a stream and just left it on overnight and it was done in the morning and you know, slowly but steadily it worked through it. Up, you know, uploading a crap load of tiles, streams are fantastic for that. I think I have about several million tiles currently switching over from between two back ends

that I set before I came out over the weekend and will probably not be done when I get back, but slowly but steadily doing that and not breaking anything. Turf is a sort of traditional GIS library for node

with sort of the node philosophy. It's written by a guy named Morgan Herlacher who got snapped up by Mapbox fairly recently and it's a less horrible version of the JavaScript topology suit. It's like pretty much the best way to think about it. So it has your sort of traditional things like buffers and centroids and convex stuff

and checking if something is clockwise, which is something that comes up a surprising amount. Jenks, and just when you have to do that traditional GIS stuff of, you know, you need to search, do a radius search in Esri server

and you realize that Esri server has no radius option with a point and you actually have to send it a circle shape. You know, something that does a quick buffer is great for that because, yeah, you really can't get that in Esri any other way.

So TileMail or Mapbox Studio, as I just got an email while waiting for this to start this presentation, announcing it. So version two of TileMail is called Mapbox Studio. So when you, don't be confused about that. It makes web maps that aren't ugly.

I mean, that's full stop. That's really, compared to Esri stuff, you can make really nice looking things. You know, the, and like, yeah. When you see nice looking maps,

they're usually made with that. And it's easier to use than you'd expect. Yeah, nothing really compares. And the, they've been introducing some vector tiles, which are some pretty exciting stuff. There's a talk next that I'm probably gonna be running away from here to go see about those. But basically you can do some very interesting stuff

when it comes to chunking up your data and transferring it in vector format with tiles. Sounds exactly what it does, exactly what it says on the tin on that. And that's gonna, for doing offline sort of data,

that's gonna be pretty interesting. So D3, we heard about that earlier, but you know, it's innovative with projections. That's not a typo. I mean, you know, I maintain a projection library and most people that deal with projections are people that have to deal with legacy data or data from legacy organizations,

like the government, who, you know, like, oh, this is in, you know, Massachusetts state plane meters. And you know, I got this, but I got this data that's in Massachusetts Island meters and you know, why does this not work? And meanwhile, D3 is, as you saw, as he talked about,

you know, it's doing stuff with, what's the projection for? Projection is for showing your map in a way that conveys what you want to show about your map and that for all the crap we give Mercator projections, you know, that's there for a reason.

There's a reason people are always using Mercator and it's not because they hate poor people. It's because it has, it's because the angles and the, you know, shapes work with Mercator and that when you zoom in in Sweden and you zoom in, you know, in North Africa,

all the angles are the same and that you can use it for getting directions. And you can sort of, there's a lot of interesting stuff you can do with projections when you realize that math is easy on computers and that you don't have to worry about on the fly projections.

And D3 is really doing a lot of that. Coop is sung by Esri. You may hear about that from several other talks, but it's Esri Server in Node.js. So that's an open source on GitHub project for using for Esri Server, but in Node.

It's pretty interesting for that organization which is somewhat like, I really feel sort of like a Hydra at some point. The different heads have very different personalities and yeah, I don't understand them all the time.

Another, so Tile Live is another technology for Mapbox. It's a tile server. If you want to serve your own tiles. So if you want to, you know, make a custom tile map and serve it, you can use that. It's relatively easy and you can, people have been building apps with it.

So Kubla is one that I wrote for serving tiles. I wrote that for code for Boston to show some historic maps I found. There's somebody who's doing Tessera, which is just sort of making tile maps. He may or may not be having a presentation next.

I don't know exactly. Somebody's doing a tile server thing then. And yeah, you can use that in sort of other apps when you just want to put in some tiles. And the last sort of thing is game. I mean, there's a lot of people that are coming into sort of spatial stuff from a different angle and game, people that are developing games,

they make, you know, they do pathfinding, but they don't call it that, or they do rooting, but they call it pathfinding and it's the same algorithms, but they just use it for, you know, units around terrain. And they do, you know, hit detection is just comparing B, that's quad trees and R trees. They're using those same things,

but they aren't using, they're in the spatial community. So I wrote an entire, like, my boss wrote an entire, like, rooting library. And then we realized that there was already one with some slightly different ways of describing stuff that somebody else had written for game, for like characters and games that have, you know, all of the algorithms.

And it's like, oh, we never thought to check. And, you know, game, you know, JavaScript games, but there's a lot of people writing games and they're writing a lot of GIS stuff without actually using GIS terms. And then sometimes, but you don't really need GIS sometimes.

If you only have points, you don't need a spatial database because a geohash works with points and it, you know, that's a whole lot easier than trying to, you know, spin up Postgres, you know. And you can do that in level DB, I mentioned earlier,

because with points, you can do that sort of linearly. And like, I think a lot of people think like, come at GIS and thinking, this is special, this is big, this is not easy. And then they have like, and then they're putting like 30 points on a map. And they're like, you're like, you don't need enterprise grade GIS, whatever.

You can just do that in memory. You know, you don't even need to index that stuff. There's only 30 of them. But that's sort of the different talk I gave yesterday. Yeah.

Questions, comments, concerns? So in your opinion, what's missing? What should the community here, what should we all be writing next that flush this out?

So should be helping with turf. So to get some of those stuff and so that turf currently relies on a, for some of the stuff JavaScript topology suite, which is just, and so getting some of those GIS stuff that your boss is gonna ask for, if you wanna switch over to node, you know, that would be helpful.

Sort of some of the spatial indexing. I've been doing some of that, but that's somewhat sparse in many ways. I mean, a lot of the stuff that isn't there

is stuff that you think about and realize you don't actually need. Like, I don't need a node GIS, full on spatial database, because usually I'll just do postgres for that, if I actually need it. I don't need anything like node specific. If somebody wants to do a pull request and connects to put in a lot of post GIS specific stuff,

that would be awesome. I have no time for that, but. Yeah. And just sort of some of the things that you wanna do that I didn't mention. Those things probably.

No more questions?