How to improve the developer experience in Heptapod/GitLab
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Untertitel |
| |
| Serientitel | ||
| Anzahl der Teile | 287 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/56993 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 2022200 / 287
2
4
6
8
12
17
21
23
31
35
37
41
44
45
46
47
50
62
65
66
67
68
71
73
81
84
85
86
90
92
94
100
102
105
111
114
115
116
117
118
121
122
124
127
131
133
135
137
139
140
141
142
145
149
150
156
164
165
167
169
170
171
172
174
176
178
180
183
184
189
190
192
194
198
205
206
207
208
210
218
220
224
225
229
230
232
235
236
238
239
240
242
243
244
245
246
249
250
253
260
262
264
267
273
274
277
282
283
287
00:00
SoftwareentwicklerDatenverwaltungCodeCoxeter-GruppeTemplateSoftwareOpen SourceDatenmodellWürfelMultiplikationRepository <Informatik>ClientNP-hartes ProblemArchitektur <Informatik>SpezialrechnerDatenbankEinfache GenauigkeitVersionsverwaltungSoftwaretestFramework <Informatik>Web-SeiteKartesische KoordinatenMereologieInternetworkingSoftwareentwicklerCodeSemantic WebÄußere Algebra eines ModulsBenutzerbeteiligungFront-End <Software>DokumentenserverOrdnung <Mathematik>Projektive EbeneCoxeter-GruppeAssoziativgesetzRechter WinkelSoftwaretestDifferenteStandardabweichungGruppenoperationZusammenhängender GraphRepository <Informatik>InformationDatenbankCASE <Informatik>ClientFitnessfunktionOpen SourceProzess <Informatik>Kontrast <Statistik>BeanspruchungWürfelVerkehrsinformationKonditionszahlTopologieWarteschlangeWissenschaftliches RechnenVerträglichkeit <Mathematik>Workstation <Musikinstrument>HochdruckEnergiedichteGanze FunktionStrömungsrichtungSoftwarePlastikkarteTermVorzeichen <Mathematik>SchießverfahrenBenutzerfreundlichkeitElement <Gruppentheorie>MultiplikationsoperatorAbstimmung <Frequenz>ResultanteWellenpaketStrömungsgleichrichterTechnische ZeichnungComputeranimation
08:19
SpezialrechnerSoftwareentwicklerTemplateMaßerweiterungLokales MinimumSoftwaretestProzess <Informatik>EreignishorizontTouchscreenProjektive EbeneStrömungsrichtungInstantiierungDifferenteSchlussregelKlassische PhysikRechter WinkelTemplateOpen SourceMAPComputeranimation
10:03
SpezialrechnerSoftwaretestSoftwareentwicklerMultiplikationsoperatorBildgebendes VerfahrenCASE <Informatik>Computeranimation
10:44
SoftwareentwicklerSpezialrechnerTemplateKonfigurationsdatenbankInverser LimesGeradeLokales MinimumZeitzoneTeilmengeDokumentenserverSoftwaretestCoxeter-GruppeBildgebendes VerfahrenFahne <Mathematik>Prozess <Informatik>Hash-AlgorithmusGebäude <Mathematik>DefaultProgramm/Quellcode
11:58
VersionsverwaltungOpen SourceSoftwareentwicklerDokumentenserverRefactoringKonfigurationsdatenbankTemplateExtranetSoftwarewartungSchlussregelUmwandlungsenthalpieProzess <Informatik>VersionsverwaltungMereologieCodeKonfigurationsdatenbankRoboterEinfache GenauigkeitOpen SourceBildgebendes VerfahrenDokumentenserverMessage-PassingMathematikInternetworkingZahlenbereichProjektive EbeneCASE <Informatik>PunktRepository <Informatik>FluidRechter WinkelDigitalisierungSoftwaretestTopologieVollständiger VerbandProdukt <Mathematik>SystemaufrufDifferenteZweiDarstellung <Mathematik>MenütechnikNeuroinformatikTrägheitsmomentIndexberechnungCAN-BusBildverstehenInformationGruppenoperationRelationentheorieNeunzehnDivergente ReiheSchreiben <Datenverarbeitung>FlächeninhaltGeradeVerknüpfungsgliedGüte der AnpassungEinflussgrößeTemplateArithmetische FolgeVerkehrsinformationVorzeichen <Mathematik>Computeranimation
19:36
GoogolComputeranimationBesprechung/Interview
20:00
Coxeter-GruppeWeg <Topologie>Arithmetisches MittelFrequenzMultiplikationsoperatorBesprechung/Interview
20:25
Web-SeiteWhiteboardInformationZentralisatorBenutzerbeteiligungBesprechung/Interview
21:12
Projektive EbeneInformationTwitter <Softwareplattform>SchedulingBesprechung/Interview
21:43
SelbstrepräsentationSchreib-Lese-KopfCoxeter-GruppeTypentheorieTemplateProgrammiergerätProjektive EbeneSoftwaretestKomplex <Algebra>Rechter WinkelRichtungInformationsspeicherungChatten <Kommunikation>Bildgebendes VerfahrenProgrammbibliothekKonfigurationsdatenbankBesprechung/Interview
23:06
Streaming <Kommunikationstechnik>Schreib-Lese-KopfProzess <Informatik>Güte der AnpassungVerschlingungBesprechung/Interview
23:53
Computeranimation
Transkript: Englisch(automatisch erzeugt)
00:10
Hi, I'm Maillou Meumont and I will present you how to improve the developer experience in GitLab and how to automate the dual works to focus on development. This presentation is more a use case presentation of how we use
00:25
Eptapod within Logilab, and Eptapod is a friendly fork of GitLab with MercReal support. During this presentation, so I will first present Logilab and how
00:40
our code is organized, and what are the problems we're facing within Logilab, what are the existing solutions inside GitHub and GitLab, and then the main part will be on the tools that we design and are open-sourced, and we use
01:03
internally to have some reviewer and some short definition and release something on the internet and how to update dependencies. But first, Logilab is a small IT company,
01:21
so around 25% in France, and it's focused on scientific computing, web thematic, and Python training, and we do a lot of custom development for clients. We work almost only with open source software, and we contribute either with code or money on the
01:43
project that we are using, and here are some examples, and on the left are some associations in France, and the right that we contribute with money, and also some examples that we contribute with code. As we are developing an application for our clients, we use
02:07
something called CubicWeb, which is a semantic web application framework developed in Python inside Logilab. You can think of it as an alternative to Django. It has an explicit
02:29
components that you can reuse from an application to another one, and we call it Cubes. There's a lot of framework. CubicWeb is front-agnostic, so you can have your back-end
02:45
written in CubicWeb, but your front page written inside a React application or whatever you want. So our text tag is composed of HTML, JavaScript, CSS for the front, front-end
03:03
Python for the back-end with CubicWeb, PostgreSQL for database service, and eventually everything is deployed with Docker. Inside Logilab, we have a lot of repository,
03:23
and we are using Mercurial, which is an alternative to Git written in Python. In order to use Mercurial, we can't use Mercurial with GitLab, so we use Eptapad, which is a friendly fork as I said earlier, and we have a multi-repository approach. So we have
03:40
one repo for CubicWeb, around 200 repositories for public cubes, some private cubes for the client project, some open source projects that are not related to CubicWeb, and also some internal repository. So we have a lot of repository to take care of, and all of them are
04:03
hosted on Eptapad. If you have only one repository, you could think that keeping the repo clean and update is simple, and having several hundred repository is hard to have
04:21
everything clean and update, but I think I tend to disagree with that. I think that both is hard, both situations are hard, because the only difference is that the problem occur faster when you have a lot of repository, and the problem that I'm
04:44
describing, I'm facing within the GitLab is the first one is easy, having a CI, so test and lint run on every commit, having reviewer on every made request, and so you don't have made requests lying around for years, have a Korean code base so that you have
05:12
high standard code everywhere, having your CI configured the right way across all the repository, having everything released properly and tagged on PyPy for cases, because we have
05:28
Python, but also as we use Docker, having everything correctly tagged, updated, and properly set up. If you come from GitHub, with GitHub Action and Probot, you can have auto-assign,
05:45
dependabot, release drafter to do a lot of things. If you come from GitLab, you have the to have someone assigned as a reviewer some information of your merge request,
06:11
but that didn't fit our use case well, and we made our own tools, and what I will do now is present you rely on a use case example, so when a new feature is introduced inside cubicweb,
06:29
and how everything, how all the tools we design help us to review the merge request, make sure
06:41
the tests are passed, the dependencies are updated, and everything is published correctly. So for example, you have a merge request on a project, so on cubicweb, a scheduled job of assignbots will assign a reviewer base and user preference.
07:09
In contrast to reviewer-relate or assignbot from GitHub, our assignbot relies on group membership, but also on some user preference, so that if I'm,
07:28
I can say okay I want to focus on reviewing a lot of work this week, I can say that I will review only like 15 merge request during the week and three per day, and then change those
07:46
values in time, and so that all, every person does not, doesn't have to have the same, doesn't have the same workload on merge request reviewing. Also assignbot will take care of,
08:08
kindly remind you to do your work on the merge request, and put a comment if the merge request is inactive for a week. Then when you have a reviewer, the first thing that
08:27
has to do, the reviewer has to do, is make sure the CI is green before anything else, and having a right CI, right job is hard to keep it up to date. So what we do is
08:43
have a GitLab CI template. GitLab CI template is a project, an open source project inside of our Eptapod instance, and it defines a lot of jobs, and within a project we will use those jobs to
09:06
declare the stage in the job that we need, and this will help us reduce the CI load, so because all the all the jobs will be customized, and so we make sure to have the right job definition.
09:24
So here there is an example of GitLab CI, classical GitLab CI, so we have Py3 and some Docker, will be also some Docker image, but also we can have some extensions, so
09:47
even though we have some shared job definition, we can always customize a job or create a new state, or create a new job, and here is an example of the Py3 job.
10:10
There are a few things to notice, first it uses a customized Docker image with all the dependency already installed, so we don't waste CI time to install everything. It also stores
10:27
some artifacts, in our case is all the deprecation running that occurs during the test, among those things. When you have your test green,
10:45
one thing that you may want is to test your application, and what you can do is build Docker image, like so when you have your major request accepted, we define another job in
11:03
inside GitLab CI template that build Docker images, and what's present here is inside the image build latest job, you are using some specific tags, so latest,
11:22
and also the hash of the commit, and we do this only if we are on the default branch, so if you are in GitLab it should be a master, but if the CI is running on a tag inside the repository, it's not the same tag to apply to the Docker image, so here we are using
11:50
the CI commit tag, and we do this only if we are on a tag. Then when the major request
12:03
is accepted, everything is published, at some point you will you want to release everything, and in our case to PyPy, so that you are using release new, and release new is so it's a small example of how it behaves when you run it,
12:27
it does a few things, it detects the new version number that you will use for the next version, and updates the code, and we do this by using the thematic versioning of all the commits
12:43
between this version and the last one, and we generate a change log from the commit message, but always let you modify it if you want, update the package, the packaging information, so in Python the manifest to manifest, and after all that it try to commit and tag it correctly,
13:08
and as we are using GitLab CI template, the next only thing you have to do is push everything to your repo, because we have another job from GitLab CI template,
13:26
saying hey if you are on a tag that shows that something that you want to publish, then run this specific this specific job that will itself publish to PyPy.
13:44
Now that you have your everything released, you want to update all the dependencies, and for that we use kubectl, kubectl can update the repository for various reasons,
14:06
the first one is pretty simple, it just rebase all the merge request in a project to make sure that we can safely review the work and merge it, and we don't have, we know if we have some
14:28
conflict, there is some conflict, and we have to do some manual rebase, manual rebase, but also it create a merge request to update the package version, so here on the right part
14:44
you have an example of a merge request created by kubectl to update kubectl to the next version, because at some point we release a new version, and inside this repo it updates the code to
15:01
update the dependencies automatically, so as a reviewer, as a maintainer of a repo, I will have merge request to update everything, and the CI will be green, and I will just have to click on the on the merge button, and everything will be updated,
15:26
but also kubectl will take care of some to create a merge request for some lint configuration, and also you can also create some rules to automatically refactor your code base
15:45
on a repo, so for example as you saw earlier, we take care to collect all the
16:04
what are the deprecation warnings occurring, writing a new rule inside kubectl to update the code, and then having kubectl creating all the merge request needed to fix
16:23
the deprecation warnings, and with that let me conclude, so the main part is within LushLab we are using Eptapod, and this is a great tool with a lot of features,
16:42
the main one such as the scheduled job to have something run every hour, every weekday, every week, from assign bot to kubectl to everything, and we are using the registry a lot, so docker image registry with docker images correctly tagged, but also some package,
17:07
python package registry, and we rely a lot of the on shared job definition. Having a good CI helps a lot to keep properly clean and update,
17:27
however having a good CI needs a lot of manual effort, at least in our case, and we end up writing some tools, so the tools that I presented you,
17:43
so assign bot to have some reviewer assigned automatically for a project, and make sure to fluid someone with tons of on tons of merge request every week. We use our own version of GitLab CI template, so we have one place, one
18:06
single source of truth to create our jobs, and we can update it to have everything have all to handcraft all the jobs, and making sure to take full advantages of the CI,
18:28
so having custom images, artifact, everything. We rely on release new to easily create a new version on the repository, so having always the same commit message to
18:49
on a new version, creating a change log, making sure the package the packaging is good, is right, and then GitLab CI template again to actually do the work to release to the internet.
19:07
And then when we have to update all the repository to update the code itself, we use kubectl with some rules that we write ourselves, and that help us to have all our
19:25
repository, several hundred of repository always clean and update. And that thank you for your attention, and see you in the question phase. Thank you for your talk. I'm looking if we have
19:51
any questions in the channel. Seems there are no questions here in the channel, but
20:05
after looking on your presentation, I was thinking how you keep track of all the merge requests. I mean because you automated pretty well everything, and how you keep track to see that the merge requests are not piling up, or if everything works fine.
20:25
Yeah, we did have a small period of time when we were having a lot of merge requests, and we're not making sure if everything was merged and reviewed, and we end up having
20:42
custom dashboard. So the dashboard, the small static pages made with pages features inside GitLab. So in GitLab you can have on every commit we can rebuild the dashboard and see all the merge requests that are open, closed, ready to be merged because the CI is green.
21:07
So we have this small web page that gather all the information. It's a central web page, or is it for each merge request? No, it's a global one. So we have a project dedicated to
21:22
gather all the information. So every hour I think it runs and schedule jobs, and it pokes around with the GitLab API to see how many merge requests there are, if they're open, closed, and everything. Cool. I'm looking at the channel if we have
21:46
any questions. No questions at all. Then let me think, after representation you said you have GitLab templates, and those templates are like... what type of templates do you have even?
22:08
I guess you have like a library of templates, right? Yeah, we have several templates, many of three kinds of templates. One based on the linking problem, making sure that we have
22:24
correctly configured for each project. For Python we could have some for our JavaScript project, some to have the test correctly configured. So in Python 3, making sure the test could
22:40
run concurrently, store the artifact of the test, and the one ready to release everything. So release to bypass, release to Docker, but the image registry inside the GitLab is also on
23:03
Docker Hub directly. Okay, we have one question in the chat. I was also wondering how you keep track. I mean, you mentioned you forked GitLab. I mean, it's a big tool, and how you manage to keep upstream? How you manage the... So Eptapod, the friendly fork of GitLab, is not managed by us. It's a small company called
23:28
Octopus. I have checked, I will send you the link just after. So it's a dedicated company that does this for us. So we send money to them saying, hey, please make sure that
23:46
you follow GitLab upstream, and do not wait two years before updating anything.