Optimizing your core application for integration
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61707 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023521 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Point cloudEmbedded systemSalem, IllinoisTask (computing)Link (knot theory)INTEGRALPhysical systemCore dumpProjective planeWeb 2.0Cartesian coordinate systemExterior algebraForcing (mathematics)System callBuildingOffice suiteExtension (kinesiology)Web browserTask (computing)Computer fileOpen sourceoutputOpen setProcess (computing)Real numberData managementInstance (computer science)Multiplication2 (number)SoftwareCollaborationismMoment (mathematics)Message passingProduct (business)Computer animation
04:52
Open setArchitectureWeb browserINTEGRALInformation securityComputer fileOpen setField (computer science)Order (biology)Client (computing)Binary fileWeb browserCross-site scriptingProjective planeIntegrated development environmentDomain nameInterface (computing)Physical systemCartesian coordinate systemInformationObject (grammar)Direction (geometry)CodeDigital rights managementSystem administratorServer (computing)DebuggerExtension (kinesiology)Installation artMobile appCore dumpComputer architectureTelecommunicationTask (computing)Multiplication signProduct (business)Matching (graph theory)Arrow of timeBitRight angleProcess (computing)Level (video gaming)Front and back endsFiber bundleDifferent (Kate Ryan album)WindowOcean currentUniform resource locatorMoment (mathematics)Revision controlFeedbackComputer animation
14:08
Token ringComputer fileConnected spaceContext awarenessInternet service providerMoment (mathematics)Uniform resource locatorConfiguration spaceRevision controlSet (mathematics)Scripting languageSystem administratorProjective planePower (physics)Different (Kate Ryan album)Relevance feedbackOpen setSurfaceCore dumpIntegrated development environmentGame controllerVulnerability (computing)Complete metric spaceINTEGRALRight angleConnectivity (graph theory)Group actionBranch (computer science)Online helpAlgorithmCodeCentralizer and normalizerPhysical systemCartesian coordinate systemIntrusion detection systemClient (computing)Vapor barrierProduct (business)Extension (kinesiology)Mobile appMultiplicationValidity (statistics)InformationInheritance (object-oriented programming)Distributed computingMedical imagingDigitizingComputer animation
23:23
Standard deviationOpen sourceAvatar (2009 film)Physical systemComputer animationLecture/Conference
24:15
Computer animationProgram flowchart
Transcript: English(auto-generated)
00:06
OK, so we're going to start the next talk. The next talk is about optimizing your core applications for integration, learnings from integrating OpenProject with NextCloud. Thank you. Hello, everyone.
00:20
My name is Willem Lindenthal. I'm one of the lucky co-founders of OpenProject, and today I'm here because I'm also the managing engineer of the OpenProject NextCloud integration. And just to get the first thing, so the most important message is out right now. I assume that here are more core developers of open source software, collaboration softwares,
00:41
and also product managers here in the room. So for you, please take home this message. Prioritize integrations on your roadmap, OK? So right now is a great moment where the public sector, not only the public sector, but many people ask for integrations, that we integrate our open source software
01:01
together to provide an alternative to the Office 365, to the Google Cloud, to Google applications, and so on. Right now, people ask us that we join together forces so together that we are stronger and become stronger in each single application so that we provide an alternative to the big techs.
01:25
OK, so integrations are really important. And with that, every single application, if we integrate, every single application doesn't need to do what the other application does. So we can focus on our own stuff and get better in our own silo,
01:41
in our own puzzle piece of the big picture. Now the bad news. Building integrations is unnecessarily hard. The good news is you always encounter the same problems again and again. So that's kind of predictable what is going to fail when you do an integration.
02:04
So my call out here is please drink your own champagne and participate as core developers in the integration projects that integrate into your system because then you see the issues that people face when integrating. Because integrations are amazing, right?
02:21
Just imagine what the web would be without a link or a browser would be without extension or a mobile phone without application. It would be boring, right? And it's just unpredictable what integrations come out if you just let people do. And people have great ideas. So the one example of such a great idea was the public sector.
02:43
The city of Cologne and the University of Duisburg-Essen approached us with the idea of, Hey, you had open project. We like you really much for organizing work, organizing tasks, organizing our teams. And we also use Nextcloud for organizing our files to share the files with our people.
03:03
And we always can find the same problem again and again. Where are the files for our tasks? So when you do some work, you do a real job. You usually need some files as input to work on the task and then at the end you need to deliver some files and put them somewhere. And you need to communicate somewhere where the files are.
03:23
So why not tracking that at the same entity, which is the ticket? We call it a work package. And solve that problem and have some real integrity between the two systems like work package and real ID linking to the files in Nextcloud.
03:40
So for the last year, roughly eight people were working on this deep integration. A lot of people, because it's pretty hard. But anyway, so what we got is, what you can see here is an open project with a work package. And you can see there's some tasks, like second fermentation dough. You want to do this, you want to have a nice pizza.
04:02
And on the right hand side you have the files tab where you see all the files that are related. And now it's not only that you have attachments, which were there already, not open project before. You can also add multiple Nextcloud instances to your open project and say, okay, these files should be stored here with that contributor.
04:23
These files should be stored there with this client, with this subcontractor. So you can have multiple Nextcloud instances connected to open project. And you will always see which files are relevant for finishing a task
04:40
and where you should put the files when you're ready and done. And you hand over the task to the next one, the next assignee, let's say, and you follow a workflow process with an open project. And on the other side, on Nextcloud, when you're on a file, or you're on a folder, you can see where these files or folders are relevant
05:02
and which tasks and which business processes they come up. So you can see here on the right hand side we have this open project tab. They can see, okay, this file is relevant in two places and two work packages. You can quickly see what status they are in so that the problem of where the files for my task are finally solved.
05:27
Now I want to get a little bit deeper into the architecture of that thing because we want to talk about problems that come with integrations. So it's a very, very high level architecture picture. As you can see, there's an open project server, there's an Nextcloud server,
05:44
there's an open project front-end that runs in the same browser that runs the Nextcloud front-end, same different browser windows. And for this integration we extended the open project core, so it's not a plug-in, it's right in the open project core which extends the open project front-end.
06:00
And on the Nextcloud side we put everything into a Nextcloud app, which is basically a bundle of code that you execute within Nextcloud. Or you could call it a plug-in or extension. And that plug-in extends the Nextcloud server and also the Nextcloud front-end. What we chose for the communication between the two systems
06:24
is that we want that every user, because users have permissions on the one system and users have permissions on the other system, we wanted to make sure that these things match up, so that an open project user that is looking for a file can only select files to link them that he or she has access to.
06:45
But at the same time we wanted to make sure that all linked files are visible to all users because someone says, this is the relevant file, maybe you don't have access, but you should know that these files exist. So we have these references by file ID from open project to Nextcloud.
07:02
And the other way around, of course as well, when you're in Nextcloud and you saw these files linked, the work packages linked, you also should only link those work packages that you have access to. So work packages are organized in projects, you're a member of a project or not, so depending on this you have access or not. So we need to have some one-on-one matching between the users.
07:22
So we use OAuth for this one-on-one matching, so users are always authenticated and don't need to re-authenticate every time they log in. But there's one thing on this architecture you can already spot, maybe, that smells like trouble.
07:42
That is the arrow in the middle that goes from the open product front directly to Nextcloud. Because that is a request, a cross-site request in the browser between two domains. And here's where trouble starts. We'll come to this later. So, clip hanger.
08:02
So yeah, from this work I just want to come up with a couple of things that are lessons learned and I would just point them out for you to think and digest a little bit what it means to your project. So, of course, if you want to have integrations, of course provide an API, that's obvious. Nextcloud, an open product, does that pretty well, I think.
08:22
We have docs and examples. But then the tiny things start. Like, okay, you have an API, you think it's cool and safe. But in the example of this direct access that I said on the architecture image, this is about direct file upload and downloads.
08:41
So this is something that is kind of obvious that you don't want to send the data through the open product server, like let's say if you want a gigabyte movie, and then to Nextcloud, you want to upload this directly or download this directly to the open project front end. So it turns out the API that Nextcloud provides for this thing
09:04
is working-ish, but not, you know, in a way that it did not work, okay? It didn't work, but it was really unfortunate because they had everything in place, but I bet that they never had the scenario
09:22
in their own company to use that in a cross-site domain environment. And so they didn't come across like twice course problems, you know, your course security on the browser that prevents you to do stuff that you shouldn't do in the browser. This is like browser security features,
09:44
and if you just have non-browser clients, these things don't show up. If you just have a Nextcloud client on your desktop, these things don't show up. They are not browser context. So my lessons learned is here, also for us, open project, we are not safe in these fields neither.
10:02
Make sure that if you provide an API, also test it in a browser setting, right, and not just write it and hope it works. You have these course pre-flights. You need to check if the course pre-flight itself doesn't fail, right? There can be issues. Okay, another one, a tiny one, but I need to say this.
10:21
You know, we as open project, we want to link files, and these files we don't reference by path because the path is different for every user or can be different for every user because if you get a file shared with you and you save it in your Nextcloud with a different file name, the path is completely different and it doesn't work for the other user. Another user, there's an open project project.
10:42
So we need to reference the file. But there's no API for getting files by ID, especially if you want to know what is the original file name that the owner of the file gave to the file, or if you want to know if the file was already deleted,
11:04
if it was trashed, it was in the trash bin. Information like this, we want to ask them frequently in order to update our knowledge about the file on open project. It wasn't there, probably never needed on the Nextcloud side, so that's cool. They don't need it, but kind of obvious,
11:20
there's the main objects of Nextcloud, and we need them, we need the information. And yeah, this problem became visible here. If you want to build, allow, if you want to allow integrations into your system while creating a plug-in interface is a great idea.
11:41
Open project has a plug-in interface, you can extend the open project code as you like. You need to be the server admin in order to make it run. Nextcloud is much ahead in that sense, they have the App Store, you just can click and from a catalog, and as a, not as a server administrator, but as an application administrator
12:00
who doesn't like to talk to server administrators, you can simply download, download and install extensions to Nextcloud, it's in the core of Nextcloud, and that's really, really cool, it's amazing, and I'm jealous, I would love to have this open project. Yeah. It's all PHP, you know.
12:22
Actually, no, the App Store, I think, is Ruby. Is it? Yeah. Okay, tomorrow it's done, no, I'm joking. It could also be Python, I just have something. Okay. Okay, thank you. I don't know how much you care about the rest, I don't think. Okay, so once you have this plug-in interface
12:42
built in your application, then the typical things that every person that is implementing this interface comes across. For example, if you want to have cross-site fetching, cross-site calls, you need to allow, in that application,
13:04
extend the current content security policy for cross-site requests, and there's something, for example, that we discovered that open project did not allow or make easy in the first place, so we had to extend that. It's a typical one, it only, you know, in the theory, everything is working, and then you run it in your browser,
13:21
and in the very last moment of implementing all the features, you see, oh, it's not working because of content security policy. Okay, there's a nice one. So when you're, like, an application administrator in open project, and you're also application administrator in NextCloud, and you want to set up the connection, the integration,
13:41
so the first thing that you do is you put in the URL of the other system, right, and you want to know if you put, or you should get some feedback if the, actually, there's, if the URL's valid, let's say you're in NextCloud, you want to know that open project is actually responding, open project is responding in the necessary version.
14:00
We don't have that. We don't expose that information. You want to know if the setup is complete. There's more about the other way, like, let's say from open project, you want to connect to NextCloud. Of course, we also want to know if our extension is installed, so would be great if you have an API endpoint
14:22
that gives you information saying, nah, it's not installed, so the validation of the URL is failing. There's something that is very, very useful for the administrators to get relevant feedback for why things is not working, because if you just put in a URL, you don't have any idea why it's not working. You need to get feedback,
14:41
and this is something I think every core developer should think of as providing us a super nice candy for developers. Another one that is, that's a topic. We're working a lot with OAuth, and setting up OAuth, I don't know if you know what,
15:01
it has basically a lot of fields, like a secret and an ID. It has the redirect URL and so on and so on. If you are a hoster or a tester, you would like to script that stuff, so you would like to have an API to set this stuff up, and it's great if you provide an API for setting up OAuth.
15:22
Also, if you have an API, you can do stuff like having one button in some setting and say disconnect integration all over the place. Delete it. I want to start again. I screwed up. I don't know what to do. I screwed up.
15:42
For this, you need an API so that you can do this. If you actually delete an OAuth connection, then please also make sure that the tokens are deleted. That was something that we discovered with NextCloud, that the tokens were still around for quite some time,
16:02
and the good thing is it came up, and we could provide tickets, and things could get solved. That's also another nice one. Just imagine you do the OAuth connection, so there's one user in NextCloud, and connects as the user in an open project,
16:23
and that user happens to be admin in an open project. Suddenly, the access as an admin comes to NextCloud. There's a lot of powers. You can create projects. You can delete users. Maybe for the integration, that is not needed, and vice versa. The open project might not need
16:40
complete administrator rights on NextCloud. There's the surface for attacks or the opening for problems to vulnerabilities by a lot. I think open project and NextCloud, we could do better in having scoped OAuth access control
17:03
so that you don't allow creating or deleting projects or that you don't allow deleting users and so on, so limiting just for the user scope actions. We can do better. We'll come.
17:22
Also, what is nice, what I really enjoyed about working with NextCloud is that they provide a design system. They have front-end components that allowed us to have more consistent UI that look and feel like the NextCloud, and also increase the development speed for us. At open project, we also are working on design system
17:41
and core UI components, so if you want to have a talk, listen to a talk about this. Parimal, where are you? Tomorrow? Same time? Yeah, okay. Have fun. Then also what is really handy for developers that develop integrations for your system
18:02
is if you have containerized development environments already at hand, so it's easier to check out your code with the latest supported versions. For example, NextCloud has multiple versions you need to support, you need to check,
18:20
you want to find out if it's still working, and of course, the current main branch to see what is actually changing in the main branch. Is my code still live or is it already dying? It's really nice to have, for example, Docker Compose files that help you setting up and lower the barrier to get started in developing an integration for your application.
18:45
Yeah, also as a bonus on this, if you can provide examples how to set up your CI quickly so that you don't need to fiddle around with that, if you just have good standard practices, how to have CI running with that and that core image
19:02
and with the plugin, that would be also very, very handy for developers to have. I think we could sneak into other integrations and we could see how they did it in NextCloud, so it wasn't such a big trouble,
19:20
but if you have a good example, that is really, really helpful. Yeah, so with all these small spots of lessons learned that we have had in the last year, I just want to send you home, let it sink in, and just remember, integrated we're stronger.
19:41
Thank you.
20:01
So any questions? Yeah. Didn't quite follow why you've got multiple versions of NextCloud. How come you need to deal with different installations? This is for the generic products, you've got different ones. So the question is why we need to deal with multiple NextCloud versions.
20:22
I think NextCloud has multiple versions and production supported, so it's like I think 23, 24, 25, maybe 22 still, I'm not sure, and they have different code, but we just provide one, currently,
20:42
we just provide one piece of code that is the NextCloud app, our NextCloud app, and we don't want to provide three different ones, so we want to make sure that in that app all versions are supported. It's different for OpenProject, OpenProject always has just one running,
21:00
one up-to-date version, so you'd always need to update to the latest version unless you want to get into trouble, so that's why you need multiple versions. More questions? Okay, thank you. Ah, Ben.
21:20
Yeah, maybe I want to bring another question to a lot of our comments. I want to add on to this that I think declarative configuration is always better than scripting, so if you can, if it's possible in the application, for example, to provide OAuth configuration statically instead of via the script, I think that's always better because it allows you to basically
21:41
the stateless deploys of your applications, including the whole OAuth context. Yeah, that's just a comment I wanted to. Yeah, so the comment was just if you can provide it as a, let's say, YAML file or something like this, a declaration, it's better than to have an API.
22:01
I'm not so sure about OAuth in that moment because then you would need to have a center when you set up both systems at the same time. You wouldn't need to have a center that decides on the OAuth client ID and client secret and make it configurable from a center. Yeah, if these IDs and secrets
22:21
there get actually generated by the applications itself because they get updated and maybe your center is not getting updated, maybe you create just a vulnerability here. We were discussing this forth and back, but we believe that next step, no project that continuously update and maybe some central distribution software
22:41
might not get touched in three years and then it's outdated and the algorithms are not secure anymore or whatever, yeah. Yeah, I'm not sure maybe you talked about it or I probably was not listening. I think you mentioned files as an example for the actual what is integrated between both systems.
23:02
Was the scope broader than that or was it primarily that? Was it any discussion because there were multiple things that could be integrated in a sense, right? Yeah, this integration basically focuses on files and folders, right, because it's about integrating digital assets
23:21
into work packages, right? So you have these links. So that's the example I chose. Actually, it would be interesting to have some standards also for API standards, why not? Let's say every system has users. Why does a user need to be represented different in every system?
23:40
Why can't we have a standard on that or the avatar or there's stuff for permissions. Why can't we have permission standard somehow for all open source vendors? I don't know. Let's think about it, but I don't see a reason why it needs to be so different. Permissions have been around for so long, yeah.
24:02
More questions? Parimal. No one asked the technical question. There was a lovely portion of pizza. Did you mean by yourself? Yes, that's my pizza. Oh, lovely pizza. Pizza Napolitana. Zealand style. Okay, thank you very much.