When building, maintaining & continuously improving a SDI isn’t enough
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43324 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Goodness of fitPresentation of a groupPrisoner's dilemmaWebsiteProcess (computing)Multiplication signTheory of everything
00:53
Staff (military)Open sourcePresentation of a groupWordSlide ruleAxiom of choiceComputer animation
01:32
Open setSource codeMachine visionOpen sourceExecution unitSoftware developerTotal S.A.Menu (computing)Red HatShift operatorDifferent (Kate Ryan album)Web 2.0Factory (trading post)Physical systemBit ratePower (physics)Computer animation
02:10
GeometryInformationFactory (trading post)Point (geometry)Data managementView (database)NeuroinformatikPhysical systemMultiplication signCausalitySoftwareRootProduct (business)Fiber (mathematics)Computer animation
03:08
Service (economics)System programmingProxy serverComponent-based software engineeringKolmogorov complexitySoftwareDifferent (Kate Ryan album)Computing platformMessage passingConnectivity (graph theory)Stack (abstract data type)2 (number)Open sourceSlide ruleInformation securityMultiplication signVideo gameProcess (computing)Confidence intervalSoftwareProxy serverPhysical systemCycle (graph theory)Computer animation
04:52
Connectivity (graph theory)SoftwareCycle (graph theory)Mechanism designSoftware testingVideo gameProgrammschleifePhysical systemMereologyData storage deviceConfiguration spaceProcedural programmingDiagram
06:13
PlanningProduct (business)Suite (music)Confluence (abstract rewriting)Process (computing)Military operationKolmogorov complexityRepository (publishing)outputSource codeRevision controlGeneric programmingVirtual realityBasis <Mathematik>TelecommunicationZoom lensInformationComputer networkSoftwareCodeDifferent (Kate Ryan album)SoftwareDialectMultiplicationBasis <Mathematik>Data managementConfluence (abstract rewriting)Complete metric spaceWebsiteShared memoryProduct (business)Repository (publishing)Message passingSoftware developerTelecommunicationRight angleMedical imagingCodeSource codeMereologyTask (computing)Spectrum (functional analysis)NeuroinformatikReading (process)Military baseDependent and independent variablesLogic gateOperator (mathematics)Ring (mathematics)Multiplication signPRINCE2Information technology consultingInformationComputer animation
09:07
Software developerCodeComputer-generated imageryInformation securityExecution unitDisintegrationSoftware testingEuclidean vectorFormal verificationWeb pageShift operatorOpen setConfiguration spaceSoftwareWindows RegistryRevision controlSource codeIntegrated development environmentInterior (topology)Repository (publishing)Continuous functionDynamic random-access memoryMetric systemStack (abstract data type)Virtual machineRevision controlBranch (computer science)Product (business)Physical systemGraphics tabletDivision (mathematics)INTEGRALFlow separationRight angleIntegrated development environmentSoftware testingMedical imagingSource codeRepository (publishing)Library (computing)Level (video gaming)Multiplication signRule of inferenceVirtual machineOrder (biology)Software developerData recoveryInformation securityPort scannerFilm editingLatent heatWeb pageProcess (computing)InformationConfiguration spaceUser interfaceSoftware industryDifferent (Kate Ryan album)TorusConnectivity (graph theory)Crash (computing)Machine visionThermodynamisches SystemGroup actionData storage deviceOperator (mathematics)Data conversionPoint (geometry)NeuroinformatikSoftwarePoint cloudResultantCodeContinuous integrationSynchronizationUnit testingServer (computing)Type theoryComputing platformComputer animation
16:20
Stack (abstract data type)Virtual machineOpen setMetric systemBlogSelf-organizationLatent heatConfiguration spaceVideo gameFormal verificationFile viewerSoftwareProof theoryLevel (video gaming)Interior (topology)Point cloudInternet service providerInformation securityCodeOpen sourceWechselseitige InformationPhysical systemFormal languageScale (map)Cartesian coordinate systemOperator (mathematics)Integrated development environmentConfiguration spaceBuildingAnalytic continuationComputing platformSolid geometrySoftwareInformationOpen sourceBlock (periodic table)BefehlsprozessorVelocitySoftware developerInterior (topology)Source codeSemiconductor memoryMultiplication signLoginVolume (thermodynamics)Programming paradigmPhysical systemProduct (business)Point cloudSmoothingHuman migrationConnectivity (graph theory)Loop (music)Module (mathematics)Response time (technology)CodeFile systemOperating systemVirtual machineMetric systemFormal languageMachine visionDifferent (Kate Ryan album)Series (mathematics)BitNeuroinformatikWeightObservational studyCASE <Informatik>Digital photographyNumberINTEGRALCovering spaceGreen's functionReading (process)MereologyCycle (graph theory)Arithmetic progressionVideo gameForcing (mathematics)Software testingThermal conductivityArrow of timeBlogComputer animation
23:34
CASE <Informatik>Product (business)2 (number)Cartesian coordinate systemComputer hardwareSoftware developerProgram slicingSet (mathematics)MereologyProcess (computing)NumberBitMultiplicationMultiplication signEquivalence relationInstance (computer science)Stack (abstract data type)Presentation of a groupDifferent (Kate Ryan album)Data miningChainInformationType theoryGrass (card game)Data modelUser interfaceWeb 2.0Descriptive statisticsGroup actionWeb applicationPhysical systemSingle-precision floating-point formatTask (computing)YouTubeCore dumpGeometryCodeSoftwareComputer animationLecture/Conference
Transcript: English(auto-generated)
00:09
Okay, now we're coming to the last presentation for today. I'm happy to see that there are still so many people. And, yeah, Emanuele Belo from KentoCamp is now going to talk about DevOps processes
00:23
to the rescue when building, maintaining, and continuously improving SCI isn't enough. Good afternoon. Thank you for being there so late. I tried to give my talk so that we finish on time today. It's a challenge to present DevOps processes in 20 minutes,
00:44
even in a larger project like the one I'm going to present. But let's try. So first, in a few words, the agenda. I'm not going to present KentoCamp. The KentoCamp presentation is in the slide. So you can have access to the material.
01:02
Then I'll summarize in a few words the project. There are two more talks tomorrow on the topic. And so I recommend you to go for the overall overview of what we have done to these talks. And then we'll dig into the DevOps aspects
01:20
of the project. So I said I would skip KentoCamp. We are an open source company. Nearly 100 employees in France, Germany, and Switzerland. We participate to open source project and propose different solutions.
01:41
Web.js, SDI, Odoo, 3D with Cesium, QGIS, and also tailored project based on OpenLayers, for example. We are also contributors to OpenLayers. And all these solutions, they can be deployed on Docker with Popet to automate the deployment of the solution
02:01
on orchestrating systems. And this is the last aspect that we used in the Fiber2Home factory project. So I'm going to jump into the project we have done with Deutsche Telekom.
02:20
Torsten Dre, the Cesium Chair, is the product owner, project manager at Deutsche Telekom for this project. And we have developed this solution with other companies like Terrestris, Mondialis, and others, you will see the slide. The goal of this project is to be able, in a very short time, to plan the roots
02:43
of the fiber network when you plan to wire a new city. So you need to collect data, you need to automate the processing of this data, and then compute the roots, and then propose possible scenario that are least cost effective,
03:02
that are interesting from a cost point of view for the planning. And so to do so, many processes need to be chained together in a special data infrastructure. So there are multiple components.
03:22
They will be presented more in detail in the following talks. But just for you to know, the challenge of this project is to bring all these components in a short time in a platform. And this is one of the challenge.
03:40
The second challenge is to work as a team. So there are many companies involved in the project. Each company has its own way to work, to develop, to deploy their software, and we have to work together on one SDI, and so this is one of the challenge. We have also different competencies,
04:00
and so we have to work together and bring all components from these different companies in the same place. And so this is the slide explaining the different components. So there are many technologies from different open source stacks that needs to be brought together
04:21
behind the security proxy, so it has to be unified in a coherent platform. So if you think of this, so many components, so many processes, such a large team or a diverse team, it's not just about develop, build, ship,
04:40
and run like a Docker message. You have to be much more structured and many more processes should be taken into account to reach this goal. So actually it looks more like the DevOps life cycle. You have to plan, define what needs to be done,
05:01
plan the work of the whole team, then you have to work together to create the solution. You have to propose a testing mechanism to be sure that your components work with other components, that the quality is there, then to package all these different components to be able to release them on a system,
05:23
on a central system that will be then configured by infrastructure engineer that will be monitored and the loops go on with the next, let's say next print, next release of the solution. One of the key aspect of this project is that at the start,
05:41
we didn't have the infrastructure we would get at the end of a project, the target infrastructure was not there yet. So we also had to build not only the software part but also the infrastructure below to run the software.
06:01
So in this talk I'm going to sequentially speak about how we solved, how we work together on each of these aspects of the DevOps life cycle. So first we have to work together to plan what will be in the next release.
06:20
We use Scrum as a method and there we brought everyone together to define what will be the outcome of a sprint and when I say everyone together, I mean the product owner that decides what will be implemented, the Scrum Master to organize, GIS consultant, but also developers
06:42
and also the operation, the infrastructure team because it has impacts what you develop, it will be needed to be hosted and new hosting paradigma will impact the software so everyone has to work together
07:00
in defining what will be done. We use therefore tools like the Atlassian suite, Confluence and JIRA and so my message here is really that it's not a project manager that we defined by himself. There were packages, it's a teamwork and as an example, the first task of the project
07:23
has been to propose a Git repository so that we can receive the source code developed by the different teams. So infrastructure aspect came very early in the site of the project. So this looks like this, a complete JIRA dashboard
07:43
with multiple cards, multiple persons involved, comments, definition of done, how do we agree on the fact that this is implemented, yes or no and so on. But the important part is it's not only about the tool here, it's the people's mind and a broad and diverse spectrum of people should be involved.
08:06
Then we created, during spring, so two weeks work together on the code, we created the software that has been selected for the first or the second for the sprint
08:20
and this has been done with remote teams, we were in different regions in Europe and so we used also tools to communicate, to share on a daily basis our status, to ask ourselves questions within the teams and also to share information. So we also organized workshops and so on. And here one of the key aspects of having
08:41
a smooth project is also the product owner being deeper available to respond to questions so that the team can progress on a daily basis without losing time. So here an example of a code sprint with everyone together and here on the right a slack image of communication happening
09:03
on a daily basis within the team. Once the software is developed, then we need some quality insurance and in such a large system we need to automate the quality insurance. So we had with GitLab a continuous integration pipeline
09:24
that would run tests on each commit or on each time we start the test. We had unit tests and also integration tests and as well end-to-end tests. So I log into the user interface, I do request and the result.
09:41
One of the aspects and the challenge here is that as a DevOps project development team and operation team needed to work together to bring their software in the CI pipeline because we need to have a common way to test the solution. We had tools, GitLab for this and we had to write the same way
10:02
the test and the way and bring every component in this pipeline and this has been, this is I think a solid foundation then for the project over the month as the scope grows. And so we proposed, the GitLab proposed a status page
10:26
and you can see here for every branch for the status of the test passes or not so that it's known information, the quality of the software.
10:42
Once the software has been developed, tested, it needs to be packaged to be deployed on at least developed on integration server so that people can also then decide to go in production. So the packaging aspect has been done with Docker and OpenShift Helm charts.
11:04
We use OpenShift in the project so every component has been put in a Docker container. We had Docker images that were generated automatically by the CI package, the continuous integration
11:20
as well as Helm charts that will define how these Docker containers are linked together once deployed on development or integration for example. And this looks like this. We have at Deutsche Telekom the Magetta trusted repository where the CI uploads every image.
11:42
There are security scans for the images there and you see that within each repository you will find the image for every branch or every tag of the solution. So we really have there the whole library of all images
12:02
that has been produced during the project and then that are also available to be put on the staging infrastructure. So once the images are ready to be deployed,
12:21
we have to bring the solution in production and this doesn't happen there are manual clicks but it works like the pipeline is extended until production and there we have set up a very advanced and high level delivery pipeline
12:44
with different Git repositories to hold the source code but also the configuration version of the different environments. So we had a Git environment for development platform, one for the integration platform
13:01
and one for production. And this way we can have an automated and still validated, manual validated process in bringing the source code to production. So just as an example,
13:21
once you do a commit on the source code in the top left chart and when you do a commit there it will open a pull request on the development environment with the last commit version so that the developer can say okay,
13:42
now I want to, I merge this commit and I update the version of my development environment. And so we can sync any commit in master on the development environment just by merging a pull request. So upgraded the version of the development environment.
14:02
And once some tests have been done there and we are happy with the solution, we can tag in the source code version and a tag in the source code will generate a pull request in the integration version to be actually,
14:20
so to actualize, to update the integration version with the latest tagged version from the project. So you bring here a feature that has been more tested and that is frozen to a certain point. And there in the integration version it's then also possible to, once some test has been run smoothly
14:42
and we decide to bring something in production, also to do a pull request, a new version from integration, a pull request to production and there we can also then say okay, let's deploy this specific version on production. And you see that only the source code is validated,
15:00
there is only one source code with commits, tags, and then we bring tags to integration and then the specific tag validated in integration can be brought to production. So we have a real pipeline without having people messing around the source code or the items.
15:26
One of the aspects of this DevOps lifecycle is that when you bring an item into production you also need to have the right, the adequate resources there, as compute resources or storage resources.
15:41
And here we use the infrastructure as code aspects and so we define, we use the cloud system and with Terraform we provision the, in the OpenStack API machine, for example, we say I would like and describe the machine types
16:01
we would like, send the order to OpenStack API and create the machine. So the source code to create the machine is documented in Terraform. And then this Terraform receive starts in Siebel to bootstrap the compute nodes and then put that with install also, metrics, security, other aspects.
16:23
So not only the application source code is versioned but also the infrastructure aspect are versioned. So this is here an example where we describe that we would like an X large machine
16:41
to be provisioned, installed with this operating system and having below, okay, the log volumes will be assigned to this device and it will be this file system type. And so you have a, the whole project lifecycle is based on the software development paradigm.
17:07
Of course, once you are in production or ready to be in production, monitoring plays a big role because you need to know if you have adequate resources, if everything is running smoothly. And here for the project we develop,
17:21
we install two things, install and configure two things, metrics with Prometheus and logs with Kibana. So we know exactly how many CPU is used on each compute node, how many, what's the memory consumption and so on. As well as we have for the application part,
17:43
the logs, so the HTTP logs that are analyzed in Kibana and we here are able to follow request errors, time, response time and so on. And this dashboard allows also to monitor the whole platform to make sure everything runs smoothly
18:02
and if there is something to adapt, this information goes in the next sprint or next lifecycle of the project. So this is really something that it's a cooperation between the development team, operation team and we have this different loop that I explained,
18:22
create, verify, package, release, configure and monitor that repeats themselves over the month of the duration of the project. We could say okay, this is very complicated, is it really needed? When I look back at what we have done in the project,
18:42
we started in fall 2017 and at the time we hadn't the targeted infrastructure. It was all the beginning of the project, we had to start and we start with what was available, the modules that were available, the technology,
19:03
what we thought was a good idea to start with and we start. And so for example, we can see that we migrated this spatial dynamic loop, it started on AWS and then we went to services, then we opened the cloud and then we started a complete cloud.
19:21
So we have free migration over this year and a half. This is only possible if you have recipes to deploy the infrastructure and the source code. Otherwise, having such a large team, so many components and to be able to install them three times, it's a huge, it will just not be possible.
19:43
And this is just an example, because we also, at the time, we migrated the orchestration system from Azure to OpenShift, because OpenShift wasn't, we had to start with something and then we changed. And then we first said, okay, we installed
20:02
a development infrastructure, then integration, and then production, but then, okay, dev int on the new platform and so on. We had a CI CD, the tests, they were built with Jenkins, but then GitLab was more interesting for the team, more performant for what we wanted,
20:20
so we changed to GitLab the CI aspects. And as the time progressed, we changed also, we updated the software stack versions, we integrated new data, we added features in the project. And so you see that the, this DevOps processes,
20:44
this paradigm, they allowed to have a continuous working software over the time. And I think this allowed the team also to create and to add new features all over the time, without having to focus too much on reinstalling
21:01
everything on a new environment. So this is our conclusion, it scales. We started at some place, and we are now at the place we would like to be, with the feature we want, and you,
21:21
it's a good idea to start with infrastructure as code and to deploy continuous integration, continuous delivery pipelines, if you plan to build a large system, maybe for a small system, it's too heavy, but if you plan a large system, it's a very valuable thing.
21:42
And it's also, if you work with multiple teams, a common language for the team, and how we gonna deploy, how we gonna deploy, and so on. So it brings the teams together. One of the takeaways I do have in this project is that open source software is modular and interoperable.
22:03
We could bring GRAS, Actinia, GeoCastra, Shagun together. It supports versatile deployment, because we deployed in so many environments, and it can be updated or graded during a short time. And also, we used the agile,
22:24
an agile method for the project, to get scrum, and then later, safe. And it empowers the team. Everyone participate to the definition of what will be done. It improve also mutual knowledge and trust. And I think it's very important,
22:40
because as the system grows, it is more structured, but also more complex. And if the teams know, the teams, every team member knows each other better, knows the subject better, the product owners know better the technique, and the technique knows better what are the goals of the project. We can keep up the speed of the delivery
23:02
and the velocity of the project. And so, for me, one of the last takeaways that such method, it allows from building blocks to build a cathedral with very different, by integrating a team of very different person,
23:21
and we can bring all this together, and bring something solid, build something solid over the month. So, if you would like to know more, there are two talks tomorrow on this topic.
23:42
One of the processing aspects with grass, and one from Tostan that will summarize the whole aspect of the project. Okay, there was a first question over here. How many people were involved in the project?
24:02
I mean, in fact, developers, product owners, so on. Maybe I can answer by myself. It's hard to tell, actually, because not all developers are working full-time on the project. As you saw, there are so many different companies involved, and not all people from the companies
24:21
are there all the time. So, in total, I would say for the geo part, I mean, the project itself is much larger, but for the geo information part, we are just counting numbers, something between 30 to 40, but I would say the core team, or if you calculate full-time equivalence,
24:40
it's maybe 10 to 14 people. In the process description, you say that there was a single scrum with single PO and so on. Is it actually so, or were there multiple teams, or how did it actually work?
25:01
I think we have a scrum method, and we adapt over the time. At first, we had one single scrum process, and then as the system grew, then we have multiple scrums in parallel, up to three.
25:22
Hi, so I actually want to catch on that last question. So, you said that you had multiple scrum teams. How did they work, essentially? So, you have multiple stacks of things that your product needs to touch on. For instance, you start maybe from hardware,
25:43
go up the chain up to, I don't know, if you take a product like web, you go development, and then web platform, and stuff like this. So, you have the horizontal slices, and then you have essentially product teams that cut vertically through the stack that you have.
26:04
So, how did you guys organize it, and how was it managed? Yeah, as Emmanuel just said, it changed over time. So, in the beginning, we just had one large team, and we were also learning to work in a scrum mode, and yeah, we rather soon realized that the team
26:23
is too big, and that we have to do something. So, now we have three teams, and it's more or less divided by topics. So, we have one team dealing with all the DevOps part, and infrastructure as code part. Then we have one team, which is mainly working on the data part, because we also have to import,
26:41
and even buy a lot of data, and organize the data, and getting data models right, and so on, and another team working on the application part. So, everything, what is needed for, yeah, geo-processing, processing the data, and also for building the web applications for the user interfaces. So, we have different teams,
27:01
but also everyone is able to ping someone else to ask questions, and to get forward in his task. All right, then I'm gonna ask a second question, I'm cheating a bit here. So, I really love what you did with the whole process, and the presentation, and I think it can apply to any type of software development,
27:21
and was there anything specific to spatial data infrastructure that made it different, or was it both a challenge, or stuff like this? I think one of the aspects is the amount of data
27:42
we do have when we collect large data sets, but it's also not specific to geospatial. I think the whole concept could be applied to any other use case, where you need a lot of hardware, and a lot of developments. It doesn't make it special, because we're dealing with the geospatial information now.
28:05
Any more questions? No? So, maybe just one more note. I mean, you already see there are two more talks, and if you're capable of German, there's also one presentation available via YouTube from a colleague of mine from Tobias Frechen, who is dealing more with the CIACD parts, technically,
28:24
which was held on the German conference, the Foscis, this year, but it's in German. Okay, thank you. Thank you, Manuel. Bye-bye.