The best of both worlds: Using open and closed source s/w to manage Transportation Networks and associated LRS
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43582 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Open sourceTransportation theory (mathematics)Likelihood-ratio testComputer networkGoodness of fitSoftwareOpen sourceProper map
00:18
Transportation theory (mathematics)Open sourceClosed setOracleLikelihood-ratio testExtension (kinesiology)Server (computing)Computer networkClient (computing)Linear mapPhysical systemElement (mathematics)Event horizonCharacteristic polynomialSign (mathematics)CodeTerm (mathematics)Server (computing)QuicksortData managementClient (computing)Point (geometry)Product (business)Process (computing)Element (mathematics)SoftwareLinearizationDifferent (Kate Ryan album)Event horizonGoodness of fitUniform resource locatorRoutingDistanceEndliche ModelltheorieNeuroinformatikOracleSheaf (mathematics)Dynamical systemCASE <Informatik>Physical systemRight angleLimit (category theory)CircleRoundness (object)LengthComputer animation
03:48
Computer networkType theoryComputer configurationLikelihood-ratio testData modelData managementGeometryAttribute grammarLogicGroup actionLinear mapState of matterHill differential equationServer (computing)Computer fileDynamical systemJava appletSoftwareWeightArchitectureWeb 2.0OracleGradientImplementationTemporal logicView (database)Parameter (computer programming)LoginGUI widgetStatement (computer science)DatabaseOpen sourceAxonometric projectionMaß <Mathematik>File formatASCIIEncryptionMechanism designDefault (computer science)Open setVolumenvisualisierungGeometryMappingTable (information)Level (video gaming)Server (computing)Attribute grammarComponent-based software engineeringOrder (biology)Web 2.0Shape (magazine)Computer architectureEncryptionData conversionComputer fileSoftwareEndliche ModelltheorieView (database)QuicksortSlide ruleRun time (program lifecycle phase)DatabaseJava appletVariable (mathematics)RoutingPhysical systemClient (computing)Configuration spaceLoginSubstitute goodSQL ServerLogicRollenbasierte ZugriffskontrolleInformation privacyState of matterLikelihood-ratio testMetadataImplementationTemporal logicGroup actionPresentation of a groupWater vaporWrapper (data mining)Execution unitRepository (publishing)Open setEvent horizonGreatest elementChainWindowProcess (computing)Right angleRevision controlGame theoryWeightFlow separation40 (number)Archaeological field surveyMultiplication signArithmetic meanInformation securityDynamical systemVideo gameInformationOffice suiteInterface (computing)Social classStandard deviationRange (statistics)Symbol tableSign (mathematics)Address spaceWeb browserCartesian coordinate systemOracleMechanism designProper mapForm (programming)LaceWebsiteMetreOpen sourceWiMAXBeer steinNeuroinformatik
12:19
Sanitary sewerGeometryServer (computing)DatabaseVertex (graph theory)Dynamic random-access memoryLocal ringStructural loadType theoryLibrary (computing)Configuration spaceTable (information)Level (video gaming)MetreSign (mathematics)Multiplication signVector spaceContext awarenessServer (computing)Open sourceQuicksortShape (magazine)Library (computing)Goodness of fitMathematicsSlide ruleSoftware developerPlug-in (computing)Different (Kate Ryan album)Revision controlCartesian coordinate systemDot productComputer fileSoftwareSocial classMeasurementOpen setLinearizationSpacetimeConfiguration spaceCore dumpBuildingJava appletOracleCNNWorkstation <Musikinstrument>Point (geometry)VideoconferencingService (economics)FlagMappingProcess (computing)Arithmetic progressionCASE <Informatik>Element (mathematics)TouchscreenEnterprise architectureRight angleInteractive televisionResultantSequelUltraviolet photoelectron spectroscopyExpressionPhysical systemGeometryForm (programming)MassWordOptical disc driveSystem callArchaeological field surveyState observerGreatest elementLibrary catalog
20:50
Lecture/Conference
Transcript: English(auto-generated)
00:07
Thank you, and good afternoon. Today I'll tell you a story. It's a love story, right? It's a true story, and it's a story between open source and proprietary source software.
00:24
And which we think does have a happy ending. And the narrators of the story, it will be Pano, that will be me, Gopi, that's the gentleman there, which is, you know, if you got any questions later, you ask him, not me, right?
00:40
And Mike, which wasn't able to make it, but I'm sure he's watching us from home. Hi, Mike. Right, so I'll start with some background, what was happening in around, you know, 2015. So we had a product that we had the server and the client software, right?
01:01
And it was a network manager, which was Oracle based, and it was an LRS engine. So how many of you know things about LRS, linear reference system, not you Seth? Apart from you, right? Okay, right, okay, good. And then we had the client with special, you know, special data manager was called, and it was an ArcMap9 extension, and it was desktop.
01:26
And so before I go, that's why I ask you about LRS. I'll just briefly background what an LRS is. So it stands for linear referencing system, and it's a method of locating something along a linear element, okay?
01:43
Usually from a known start point. And in our case, in roads, the linear element was invariably, you know, a piece of the road, and the event can be anything that's on that road. And events can be physical items like, you know, signpost, like, you know, safety fence, all that, a road characteristic, like, you know, speed limit, or it can be like any true event, like, you know, like a traffic accident.
02:05
So the location, that's like LRS 101. Location is the distance between the start point and the event, okay? And in a graphic sort of way, you know, this is a small road, and it starts usually here with an intersection, and it ends there, another intersection, and it has a name.
02:25
That's the way that we modulate it in the system, and it has a name or code, and it has a length, okay? Which means then that if a signpost is 1.8 miles along that road, A123, it's around here, okay? And there's a computing process used to locate items along, you know, linear articles, and it's called dynamic segmentation.
02:46
I'm just mentioning it here, you don't really care. Then usually you locate things along routes or routes for the Americans, or some Americans. Anyway, and so, which is made up of lots of different segments, okay, of roads and intersections.
03:05
And that may, and that's where the modeling sort of challenges start, is that they may not start at zero, okay, might not start at zero or at the end or start of the intersections that may overlap or may have gaps, okay?
03:26
Or maybe circular, which means that you don't know where the start is, okay, if you want to locate some start or end, okay? And events may also be located not just from the start of a route, but also from other events, okay?
03:41
So this is what we have two different linear location methods for our island systems, and sometimes they're used in the LRMLRS or used interchangeably. Again, not something that you need to worry. But also, another challenge is what happens when you have all your mileposts, you know, at the correct mileage, and then, you know, there's a road realignment, and a new housing state is built, and then all your mileposts are just wrong, okay?
04:08
Now, our LRS engine, our LRS engine deals with all that, okay? And more to that, like you can handle any network, there's a flexible network model, it's not just for roads,
04:21
you can have like rails, you can have like water networks, you can support multiple LRSs, you can have, you know, it's user-configurable at name and attribution, so you can name your network, and, you know, you can use any name convention that you want, and you can also model the datums with the smallest unit on the network, but also linear chain datums that make up routes, or non-linear, sort of just a bunch of buckets of datums,
04:48
or even that you make a group, or even a group of groups, so you can have like admin-type groupings, okay? You can also have temporal management, you know, the full network history is maintained,
05:01
you don't delete things in our system, nothing gets deleted, just end dated, so which means that you can undo edits at any time, okay? And it supports all the circular routes, the Gaussian routes, and another thing that separates from other like software is that it's a logical first approach, so it's the logical model that we deal with mainly, and also it has a geometry as an attribute,
05:25
so we just delete, the geometry is just an attribute of the network, rather than what defines the network. Let's make it really, really fast. So, it was all good, but we had, you know, problems with the UI, okay? It was a desktop, it was dependent on Esri technology, and, you know, interoperability in general was an issue.
05:46
So what we wanted was to be web-based, and to be open, as in open data, open standards, and to remove any Esri dependencies. So this guy comes along, and he's like, you know, what if I told you?
06:02
Well, he didn't do that, but, you know, he could have, but, you know, but basically we discovered open source, right? We discovered, that was the love at first sight thing, and, because we used, you know, open layers, and we used open layers where we switched to all three immediately after release, and we're currently working on, working with open layers four,
06:24
and we're looking to move to open layers five. And we also use maps, so those are the main two components, if you like, you know, the 4G components that we use. So open layers and maps server. And it's, we use server-side, you know, special publishing. I've got this item here, which I think I've put it on a different, another presentation I did,
06:44
you know, why didn't you use GeoServer? Because we didn't know enough Java, right? And it was a Java thing. Well, mainly, one of the reasons, also it's quite, you know, it's more heavy, you had to, it has its own login system and all that. But, you know, that's it, that's all I know, no one shoots me down, right?
07:00
Okay, so, but, you know, but also, you know, Bentley is basically, you know, by maps server you can use C sharp interfaces, and Bentley is usually, it's mainly a Microsoft shop, okay? And with maps server, we used dynamically generate map files, and I'll explain this next slide what I mean by that.
07:20
So, and this is the architecture in a sort of, you know, very high-level architecture. So, you know, the left-hand side, can I use that? Maybe not. Okay, but on the left-hand side, you can see, you know, client, client browser, we've got open layers, we started with using Angular, now we're moving slowly to ReactJS, we got a sort of, you know,
07:42
a ReactJS wrapper against Angular, you know, we'll move full Angular. Eventually then, in the middle tier, we got the application, it's classic, you know, 3-tier architecture, you got the web API, at the bottom you got, you know, the maps server that will create the map files, you know, from the database server, and it's mainly Oracle, as I said, but we're looking at migrating as well
08:03
to SQL server as well, Microsoft. And, so, what do I mean by dynamically generated map files? Because we're Oracle, and that's one of the things that we want to get move away from as well, this is Java, we don't like Java,
08:21
we started with MapBuilder, so MapBuilder is an Oracle Java application, that all the Oracle special tables, they have special metadata tables in Oracle, where you can set styles and labeling and all that sort of thing. And then, so once you have that, then we have like a conversion mechanism, if you like,
08:42
so as soon as the user logs in, and because we got also this, you know, role-based security, basically creates a map file for each user, dynamically, so that's what the dynamically generated map files means, okay? And, you know, one of the reasons we want to get away from that is that
09:00
now we're using, obviously we can display Oracle, you know, special tables as layers, but in order to get, you know, shape files and anything else that that map server supports, we have like, you know, config files, you know, that say, okay, I got this shape file, and I got this rusted data, and I got this background maps, and we want to get away from all that.
09:22
And, so basically, again, what, you know, it says when you log in through a PL-SQL API, it, you know, it reads the map file, the map, build the map definitions, the tables, and then it combined with the, you know, custom Bentley layer metadata, and then allows implementation for role-based security.
09:41
And what we also did, you know, we got views, you can have, you know, implementation of temporal views, so, you know, it's not just what the user can see, but we'll see only active data, because everything else will be end dated. And we also use, you know, runtime substitution variables within the map file, so we can use, we have like, you know, quite clever thing doing about, you know,
10:01
having lateral offsets, which basically offsets linear, linear offset event data of the route. Map server file, people are aware of map file, map server, what they are, yeah? Okay, not you, Seth, again. But, you know, it's a map server file, it's an ASCII file that contains definitions for the map container extend,
10:22
it's a very, very, sort of, you know, again, simplified view. So, you got the map container extends, then you got, you can define the symbols, the layers, and all that, and it looks, so, the generate map file looks something like that. Okay? So, you can see that, you know, we got the web metadata, and all the WMS request, the WMS WFS request,
10:44
and that's what a layer looks like, okay? And you can see that here, it's like your normal, what you see, the WMS metadata kind of keywords, but also some of our own keywords, okay? So, this is, and remember, again, this is a map file per user, okay?
11:00
Which gets deleted afterwards, and whatever. So, you know, it knows exactly, so if it's editable, that for another user might not be editable, and all that. So, we, you know, we get everything, all the business logic within the map file as well. Okay. So, some of the things that we had to change and do with Map Server.
11:22
The first thing that, you know, well, they told me that, you know, we need stronger encryption in Map Server. So, because that MS encrypt, you know, was not secure enough. So, what we did, and this is Windows only, we had a new build of Map Server, where we used the data protection API,
11:41
DB API, if anyone knows about it. And that, this is from a fork Map Server, and I think, actually, this one has been committed back to the, I'm not sure, but, you know, it's there on the repositories. The other thing is, we're Bentley, right? We're Bentley, and we're supposed to be like the DGN guys,
12:04
that, you know, we can deal with DGN. Problem was, Map Server wasn't supporting DGN V8, the latest DGN format, okay? That would be embarrassing. So, you know, let's talk about V7. So, what we did, we used, because it was easy for us, we used the proprietary Tiger libraries, again,
12:21
within a build, a custom build of Map Server, okay? And now, we can support our own format, which is great. Another thing we did, and that was open layers, mainly, and with maps, I'm not sure if it's a Map Server problem, a WFS problem, all that, but basically, what we wanted, you know, it's a linear reference system, so, you know,
12:41
you click somewhere and just want to see the offset, you know, the measure value along that line, right? So, the problem was that the, and when you do that, the layer is a W, it's a JSON or a GML layer, it's vector data, right? The problem was that the coordinates that we're getting back were just X, Y's, never M's.
13:01
Map Server would just never get either as a JSON or a GML, wouldn't get the M values. So, what we did is, like, every time you click on it, and it's actually quite quick, you know, I thought it's fast, it won't be, but, you know, copy did it. We fetched, you know, measures from the, so as soon as you click, you got the X, Y's, and then you just go back to the database and get the measure value, okay?
13:22
I'll show you some examples as well. The other thing that we used to do is that we want to trace, so we create new networks, right? New pieces of network, and sometimes, invariably, you want to have, you know, you got your shape file or something else that you, you know, put on your map, and then you want to trace along that shape file, right?
13:43
The problem was, with open layers, the snapping agents and the draw agents and whatever, you know, they didn't support interactions, you know, the tracing until at the first vertex, I will say, not the whole geometry, okay? So, what we had to do is have, you know, custom draw method that we also call private,
14:01
and I know it's a bad thing, it is a bad story, guys, you know, so we call private methods, private open layers methods. And so, you end up with this, right? You can see that, you know, the mouse goes up, up and down, doesn't, you know, it gets locked onto that element, right? And then, as soon as you go back and forth, you can see, you go from here, you can click there, and it will just go there, right?
14:21
And then you can go, yeah, it gets locked, it goes through here, and it will just automatically get the trace, and then it will return back. Okay, so those are the, you know, showing, because I think they are just quite cool things. And, you know, one of the other sort of small things, again, shape files, you know, to add the shape file, you shape the JSON format,
14:42
that wasn't supporting polyline M values, again, so we just did a small change. That was a whole thing, and a great thing, and I think that's why our developers loved the open source stuff, because they can actually have the problem, you know, we'll just go and fix it. Okay, so, and that's what it looks like.
15:01
I thought I wouldn't do a video, you know, I'll just show you some screenshots. So, this is what the AEW asset wise linear referencing service, you can see at the bottom, sorry, the top left, is. So, this is what it looks like. This is your catalog, where you got all your different map layers, and you can see you got, you know, sign reads, like an asset, and you go to parking,
15:22
another as a linear asset, and then you got the federal functional class, and you get all the symbology that, if you remember, that comes from a map builder all through to, you know, a map server, and through the map file, okay? So, you have all different layers there, as well as background layers.
15:41
Now, we, the whole editing stuff, because there's a, okay, the whole editing stuff is just, it's like a start editing, stop editing. So, I have clicked on start editing, as when I did the start editing, it goes and has to go to stop editing now, it will download, it will allow,
16:00
it will ask me which layer do I want to edit, to edit this, yeah, this is a datum layer, it will download this vector data, and then you got that cool little radial menu, which is context sensitive, depending on the layer that you're editing, okay? So, it will be a different radial menu, it's an asset, it's an event. Okay, and this is just like, you know, advanced search,
16:22
you know, how you can do an advanced search on a layer type, as well as on the network. So, I will find sign reads that, you know, sign reads that's got butterfly along that network. Okay, very quickly then, what's ahead? A lot of things are ahead, but you know, what might interest you
16:40
is that we're trying to do a map server configuration tool, as I said in the beginning, we want to deprecate map builder, you know, the Oracle sort of Java stuff, and that can be used across all our apps, and basically it's another map server, you know, yet and again, another map file configuration, sort of, you know, but mainly for our purposes,
17:00
and it would look, this is just the screenshots, it would look something like this, okay? So, we can have your map, you got all your layers for your map, and then for each layer, you can have, you know, the basic, the layer name, transparency, blah, blah, the labeling, the symbology, you can see there, but that at the end of the day will go and add entries to tables, right?
17:23
And then the application, any application can come and create the map file. So, just labeling what it look like, and you can always like, you know, just use map file expressions, and that's what the class definition would look like, okay?
17:41
Again, very much work in progress, but, you know, these are the mockups, basically, that the developer working out of. And I think that's that. Questions to Gopi? Thank you. That was good on time, wasn't it?
18:03
At some point, I think it was around slide 25 or so, yeah, because I noticed. You said that you had to use a restricted license plugin for Map Server, because they didn't...
18:21
Yes, did you consider, instead of using that, extend the current support, so the Map Server supports version 8? Yes, we did consider that. We're still considering this. The thing is that, as like always, we need it now, and they just think,
18:41
like, get the library in and, you know, just do it. But yes, we are considering, you know, yeah. We already have put, you know, things in Map Server, so, you know, we have done some enhanced Map Server core. Maybe that, you know, once they have time, they'll do the next thing. It will be this.
19:01
You said you want to move away from Oracle towards MySQL or MS-SQL. Are you considering Postgres postures as well, or is there any reason for doing only MS-SQL? Thank you. Or even MySQL space shell? MySQL. MySQL, yeah. Well, look, no, we're not...
19:22
We have considered, you know, Postgres as well. The thing is we do not want to deprecate Oracle. We want to deprecate Oracle Map Builder. We won't deprecate Oracle anyway, because we already got a lot of users that are using Oracle Enterprise. They got a lot of APIs and things that, you know, working on, but the other thing that our users have,
19:43
and, you know, at least DOTs and big sort of, you know, highways agencies, is SQL Server. None of them has a Postgres, so we never been asked to do Postgres, so, yeah, no. Yes, yes.
20:01
Anyone else? We're good on time, so we can take more questions, so... Anybody? Yep. It's an easy one. You said about when you have the linear referencing and you have the history and you have something changing. How are you storing the history? Is it using Oracle table versioning? And are you going to do the same thing in MS SQL Server?
20:22
Or is it your own custom tables for history? It's custom tables history. It's basically anything. If you end date something, you get a new copy of that, and then the other thing gets end dated. Yeah.
20:41
No more questions? So in that case, we will wait a few minutes for the next talk, and thank you very much.