Turning an Old Ship Towards FOSS4G - Why, How and Where To
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68560 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
OracleServer (computing)FingerprintInformationSoftwareOpen sourceDynamical systemVariety (linguistics)MappingMereologyRelational databaseString (computer science)MathematicsService (economics)Physical systemRootLocal ringProduct (business)Scripting languageSlide ruleComponent-based software engineeringServer (computing)Computer-aided designWorkstation <Musikinstrument>Independence (probability theory)Moment (mathematics)Axiom of choiceWindowDatabaseFunctional (mathematics)Procedural programmingSoftwarePlug-in (computing)Multiplication signOpen sourceAutomationFluid staticsClient (computing)Staff (military)Line (geometry)Table (information)Lecture/ConferenceComputer animation
08:01
OraclePhysical systemSoftwareOpen sourceServer (computing)Shift operatorWeb pageEuler anglesUsabilityChemical equationScripting languageDatabaseDigital rights managementSoftwarePhysical systemMultiplication signDatabase1 (number)Euler anglesDecision theoryOrder (biology)Projective planeGoodness of fitPoint (geometry)Exterior algebraProcess (computing)Software developerMathematicsProcedural programmingPlug-in (computing)Group actionScripting languageChemical equationComputer clusterClient (computing)Functional (mathematics)Integrated development environmentDivisorUsabilityBitOpen sourceShift operatorComputer animation
15:57
OracleNeuroinformatikUsabilityMultiplication signWeightPhysical systemSoftwareFile formatDataflowClient (computing)Open sourceProduct (business)Moment (mathematics)MappingTransformation (genetics)Water vaporExpected valueInformation securityMathematicsFrustrationService (economics)Process (computing)Metropolitan area networkDatabaseMereologyParallel portDifferent (Kate Ryan album)Function (mathematics)Revision controlGoodness of fitTable (information)Computer animationLecture/ConferenceMeeting/Interview
23:53
Least squaresComputer-assisted translationElectronic data interchangeLecture/ConferenceComputer animation
Transcript: English(auto-generated)
00:00
Thank you. So welcome to the second session of FOSS4G. It is the most interesting one, I think. So you made a great choice coming here. And let's start with a person from the biggest mapping
00:21
company in Estonia. You have seen a lot of maps from them coming here, including maps that are sold at the merch table. Downstairs. So this company has been working as proprietary staff for a long time.
00:40
And in Estonia, a lot of cartographers go through the company. So they are finally turning to FOSS4G. And Pugh Palma will tell us how it goes. OK.
01:00
So thank you a lot for this introduction. I will just clear out the pronunciation. So my name is Pugh. And first of all, I would really like to express my gratitude to everybody who is involved in organizing this event. Because I personally find it very exciting
01:24
that this kind of international GIS conference is happening here in Tartu. Especially, of course, because I consider myself also a part of this local GIS community, as I have also studied in Tartu University.
01:43
And also, the company that I represent has its roots inside Tartu University and is a local company. So this is a private company called Regio. And it started now more than 30 years ago as a spin-off cartography company.
02:02
And now it has turned into one of the biggest GIS companies in all Baltic states, offering a variety of products and services based on local Baltic geospatial data and overall GIS expertise.
02:21
But I would really have to clarify that I wouldn't want you to get an impression that Regio is a big company. It's actually a very small one. We have only 30 employees. And I can only say that Regio is one of the biggest because it's also one of the earliest. And there are just not that many companies
02:44
in the Baltics, private companies, in this GIS industry. And I think it would be also fair to say that in Estonia, in all Baltics, this GIS industry is more dominated by the public sector, at least when leaving all these giant international
03:04
corporations aside. But I did not come here to talk about the market situation. I came here to share my experiences in turning a GIS system that is mainly relying on corporate software into a one that is more
03:24
relying on open source software. But before I can talk about why and how, I will actually have to introduce the system itself and the background of it. So OK, I forgot to show this slide earlier.
03:43
But anyway, when Regio started more than 30 years ago, it was actually all manual and no software involved at all. But very soon, Regio started digitizing the data and automating the workflows.
04:02
And first, it was into a CAD system using Micro Station. And later on, this CAD system was transformed into a relational database. And well, it took time. But somewhere in the middle or end of the 90s,
04:21
it settled down on Oracle Spatial, the relational database. And as a main window into the data itself, the desktop software of MapInfo was taken in use. But because MapInfo did not satisfy
04:41
all the needs of cartographers, especially when we're talking about paper maps, then another desktop software were taken in use, which was ArcMap. And so that ArcMap could actually communicate with Oracle. Also, some components of the ArcGIS server
05:02
were added to the system. And then, of course, Regio started extending these functionalities of these softwares by all kind of procedures and scripts and plug-ins to further automate the workflows, which are also
05:23
very important strings that should be considered. But even more numerous are the strings that are external of this GIS system. And by this, I mean all the strings coming in.
05:42
And we see this process, and I think it is still ongoing in all three Baltic states, that more and more GIS data sources are becoming available, either free or with reasonable cost. So the purpose of this GIS system is now more about not producing the data on the site,
06:03
but rather ingesting and finding all these data sources out there, then standardizing them and correcting them and enriching them with each other, and then serving them out to our clients
06:23
the way they need it. So is it either static maps or dynamic maps or API services or just the extracts of the database? But these are all these strings that really have to be taken account when you do any kind of changes in the database.
06:44
So this pretty much covers the GIS system before the changes. And now, about a year ago, Regio decided to replace some of the components of this system. And namely, this Oracle database engine
07:03
with PostgreSQL, with PostGIS extension, and MapInfo with QGIS, and also these parts of S3 products that are not related to the papermap production line. Because this papermap production line is,
07:22
I guess, this most conservative part of our business, so we didn't want to do very big changes in this part at the moment. And I guess now I can turn to the why part. So why did Regio decide to do that? And I really have to explain the background here again.
07:42
Because Regio is involved in a lot of projects, in some of these projects, we had this opportunity to build up an entirely independent GIS system that had nothing to do with the one that I just explained here. So we could, and as usual, it had to be done
08:03
with as low costs as possible, or no costs. So obviously we're looking for alternatives. And at that time, already this tandem of QGIS in corporation with Bosque SQL were getting quite popular,
08:22
so we thought we'll also try it out. And the first projects were very successful, and it turned out to be really reliable and convenient and flexible system to use. So what happened was that whenever other projects
08:40
came into our company, we always used the same tandem of softwares, when they were not related to this secure database that we're holding. And what happened also consequently was that kind of shift of competencies inside the company.
09:01
First of all, the people who came to work for these new projects, well, they never learned the old system. So their skills developed in this environment of this open source software. And maybe even more importantly, the people who were involved in both the old system
09:22
and these new projects, they started noticing the benefits of these open source alternatives. And by this I mean, you know, the ease of use and the speed and the functionality. And they started approaching our old system
09:40
with these new software tools. And taking QGIS to approach data in Oracle database, and also in some cases, even exporting data out from Oracle to PostGIS and then doing these processing in PostGIS and then exporting
10:02
data back to Oracle again. And when this starting happening more and more, it was getting kind of obvious that the times of this old system are getting over. And so you could actually say that at first it was because of the license fees,
10:22
but the actual decision to also turn the old system was more related about the preferences of the users. So I would say it was more about people. And maybe one thing I have been noticing about people, I would maybe like you to take this
10:41
with a bit of filter here, because it's not a scientifically proven factor or anything, but I would just have this impression that the people who prefer using free and open source software also tend to be maybe more open-minded and also more enthusiastic
11:03
about the job they're doing. And also have more of this, yes, it can be done attitude. Maybe they don't know how to do it at the moment, but they say, I'll go and look around and I'll find a way to do it. And this is, I think, backed up by this very big
11:20
and friendly community of these open source users who are willing to share their experiences. But on the other hand, the people who tend to prefer these corporate software, they tend to have more of this attitude that, well, no, it really can't be done. Or maybe it can be done with the next update,
11:43
but probably not. Or even worse, it shouldn't be done because why should we change anything? It has been done this way all the time. And I guess you have to have both attitudes inside the company, the ones that are really eager to do everything very quickly
12:00
and the ones who really question the reasonability of doing these things. But I'm pretty sure that it's essential to have these open-minded people in your company. So I think this pretty much covers the why part. So I'll go to the how part.
12:25
Well, I have to say that it has been quite a challenge. And it took us a lot more time than we initially thought. But on the other hand, we also achieved a lot more than we initially planned.
12:41
And I think we were really naive in the beginning thinking that we can transform the system as it is. So it sounded like a great idea. Let's just transform the system and not enhance anything in order to do it quickly. But we thought that maybe even AI could do it.
13:02
How hard can it be? It's pretty much based on SQL everywhere. But we couldn't rely on AI, certainly. We had to rely on our database engineers who really had to dig deeply inside the system in order to guarantee everything will work afterwards. And when you are a good database engineer
13:20
and you are kind of hands-in the system and the system is old, you will probably see things that could be changed and should be changed. So we came back to this decision point again and again, whether we should do these changes right away or not do them at all or postpone them.
13:41
And often we decided that, well, we have the hands dirty already, so let's do it. So this is why it took us a lot longer. Another thing that really characterizes this process was this kind of balancing between keeping things on the desktop side or the database side.
14:04
And our previous system was very much tilted towards the desktop side. This means that a lot of developments were done using the tools available in the desktop software.
14:21
But very soon we learned that when we were able to transform these plugins or scripts into a procedures of the database, this will speed up things. For example, we had some processes that took us before the transition, almost days,
14:44
but after the transition we could count the time in minutes, so it was a drastic improvement. So we tried as much as possible to move this balance to the other side, to the database side.
15:00
And for this, this really good relationship between QGIS and PostgreSQL was very essential. So we have in QGIS these tools to approach directly the database, and also, for example, we could also generate plugins that call out procedures
15:25
in the database, so for the end user it's just a button in the user interface, but actually everything is happening back in the database. So we're just hoping that we didn't tilt it too much
15:41
to the other side now. And one other thing I would really want to talk here about that made it complicated. We had to do everything gradually, step by step. We had to keep this system running all the time, so our clients were expecting everything to work as it has been. They expected, of course, nothing to change,
16:03
so we had to keep the system running, the data coming in and going out all the time. So for that we introduced these two ways, data flow between the databases, so we could transform just part of the database into the new system, but then when we needed actually
16:22
to use these processes that are still related to the old system, we could export the data back and use these, so we could do this transformation one by one, and actually we cannot get rid of Oracle for good, because also we have even one client
16:43
that needs as an output this Oracle dump, so we will have to introduce also this little version of Oracle to our new system. But this was really good, because we had all the time this kind of safety network behind us. Okay, when it doesn't happen in the new system,
17:02
we can always export the data back to the old system and do it in the old way, and we did have to use this sometimes. And so I have very limited time to talk about the where to part, and I do have two thoughts here. First of all, I don't think it is,
17:22
that it is, can be a goal itself becoming free and open source 100%, and we are using these solutions because we find them good, and they really fit our needs
17:40
and we are kind of feeling ourselves very good in these, I would call, open waters at the moment. But whenever that would emerge, some other source that would maybe fit better our needs, then maybe we will have to do these transformations again, you never know what the future brings.
18:01
And this brings me to the final thought, is that a GIS system can actually never be ready. I think a good GIS system is always in kind of evolvement, and so maybe a good parallel here would be our capital city, Tallinn.
18:21
We have this very nice legend about Tallinn. Besides the city of Tallinn, there is a huge lake, and inside this lake there lives an old grumpy man called Ulaam Estevanakke. And whenever somebody would say that Tallinn is now ready, then he would flush the city away. So I guess it's maybe the same thing with a GIS system,
18:44
whenever anybody would say that now this is ready, there's no more changes we can do here, everything's perfect, then this will be also the end of the system. Okay, that's all, thank you.
19:10
Anybody has any questions? Yeah, I can.
19:22
Hi, Sanne from GISP of Finland. Would you do something differently? Or was it the best way to do it? Yes, we would do a lot of things differently. We would plan the works different way.
19:41
Well, it was a bit frustrating because we had the higher expectations about the time it will take. So we would be more realistic about it, and thinking that, okay, we didn't know at the beginning that we will have to build up this two-way flow and the safety net thing, but now we would already know it,
20:01
and we would start maybe building this up, this system, and then take it really slowly, gradually. So, yeah, we would do things differently, certainly. I have a question. You have told about switching your infrastructure
20:21
on the internal side, but Rego is most known for its products, for printed maps, for online maps. So did this switch inspire some new products from Rego?
20:44
Yes, because during this process, we saw possibilities to do things quicker. So, yes, we can now offer, for example, for our clients, more dynamic outcomes
21:01
than we couldn't provide in the older system. So it's not a product on the table I can talk about, but, yes, we have this ability now to provide better services to our clients. So, yes.
21:25
Hello, Thijs Oestreijs from the province of South Holland from the Netherlands. I have two questions, actually. So the first question is, do you store your databases on-prem, on location, or in the cloud? They are actually hosted locally.
21:42
Locally, okay. And my second question is, did the move to open source give any security issues as well? Or not that much, since it's locally anyway. Well, I'm not sure about this.
22:02
Well, I guess there's no big changes in these security matters. So, first of all, thanks for the presentation. Really insightful.
22:21
I could imagine that probably everything went properly smooth, the transition, but the reality is probably that there were some setbacks. What were maybe some setbacks for you, or some concerns that maybe came up during the process? We had a lot of setbacks all the time,
22:42
because, well, we didn't know everything. Not that we know right now, but they were minor ones, that everything was, we haven't had any problem that was not something that we cannot deal with. But during this process, we could not foresee everything.
23:02
For example, we had this thing that before we did some data extracts through the MapInfo system using the format of tab, but you know that this format of tab really generalizes the coordinates, and so we were really struggling
23:21
at finding a way to generalize the coordinates in exactly the same way as MapInfo did it. We had to kind of fake it, so that our clients would, they expected nothing to change. They wanted everything to work exactly the same way it was before. We thought that now they will get better data,
23:44
more precise and things, but no, no. Okay, thank you, Pio, for this talk.