Sharing Reproducible Results in a Container
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47289 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Multiplication signContext awarenessSystem administratorBuildingComputer animation
00:24
Presentation of a groupResultantComputer animation
00:47
Boss CorporationServer (computing)Web applicationOctaveDirectory serviceComputer fileInternetworkingTotal S.A.Functional (mathematics)User profileData managementGrass (card game)Physical systemOperations researchOperating systemMultiplication signOctaveWeb 2.0SoftwareWebsiteMobile appComputer animation
02:13
Line (geometry)OctaveScripting languageComputer animation
02:29
OctaveRevision controlComputer animation
02:39
Graph (mathematics)Lambda calculusOctaveRevision controlEntire functionService (economics)Operating systemMathematicsBootingConfiguration spaceClient (computing)SoftwareComputer fileoutputSource code
03:41
Hill differential equationPhysical systemComputer networkIntegrated development environmentExecution unitInstallation artWeb applicationLambda calculusPhysical systemScripting languageComputer fileMultiplication signGroup actionRevision controlConfiguration spaceOperating systemOctaveWeb applicationSoftwareSlide ruleBuildingPoint (geometry)RoutingGraph (mathematics)Electric generatorElectronic mailing listDirectory serviceService (economics)CASE <Informatik>Type theory
09:37
Open setPoint cloudFacebookOpen source
Transcript: English(auto-generated)
00:05
It's time for the next lightning talk. Efraim is talking about reproducible containers in the HPC context. Thank you. Okay. So, yes, I came into this really more
00:22
from the sysadmin side. I like making things build. I guess I work with the scientists, and so they have their software and need to put it out there for the papers, for other things that they can go and point to and say,
00:42
here it is, here you can see my results. So I was home working, and so we got a new app that we were gonna go and host on the website. It was being moved from old one to new one. So the old website was running on CentOS 5
01:02
with PHP 5.3, with Python 2.7, and another Python from the user's home directory, and Octave also from 2010, and just everything was from 2010. Just looking and thinking. Well tested, of course.
01:21
And it also accepts arbitrary files, upload whatever you want to it. Clearly exactly what we want running. So actually, what about it using Geeks? So with Geeks, you tell it exactly what packages go into building each package.
01:42
So you get the reproducibility in that. It works also when you have a collection of packages. Every time you install it, you get the exact same packages. You don't have cruft from installing and removing packages, and it also extends into a whole operating system or in this case, into a container.
02:01
You tell Geeks, I want exactly this inside the container, and that's it. That's exactly what you have inside. So for the web app, we have all of our old software here, and I guess going back to the container, one of the scripts I had and actually patched all the way.
02:21
Just missing, remove, rmdur. Just, I didn't tell it where to find it. I didn't put it in the container. It just wasn't there. So yeah, so had old versions of Graphviz, of Octave, all of this coming down to our final package.
02:40
So with Geeks, it was easy to go ahead and to create the old versions of the packages. We started with the version of Graphviz that was already in Geeks. We changed a couple things, mostly downgrading the inputs to older versions of them.
03:01
Octave was basically the same thing. We took Octave. Q-Haul had moved around since then. Gnu-Lib had moved around since then. So just some changes here and there to make everything actually build with older versions of everything. And then for the actual container,
03:21
and pretty much this was the entire config file for the container, it's to find the operating system. Bootloader doesn't matter, it's a container. There's no firmware, no packages, just the actual service itself and the DHCP client so it can actually find the network that you give it.
03:41
So the service gets fed a service type, a service configuration, which in our case was, can't see the mouse. So here we just have the actual package that I built at the end. I told it that it was going to be in the serve HTTP directory with that port.
04:02
Our little activation script here really just says, before you start, go ahead and run these actions so it was, if the directory exists, delete it, then recreate it, copy the stuff over and it wants to write to the directory so go ahead and make that available.
04:22
And the whole thing was just behind nginx so with the system management, the config for nginx, it was really just listed on the port that we had on the previous slide using the route that we had on the previous slide from the configuration. Then service type down here was the one
04:41
that we actually passed to the operating system config to make the container. We said, when you activation service type, when you start, run the script, you're using nginx with the config that we have and by the way, don't make sure that PHP FPM is running.
05:04
Building the container was the system container command, you feed it the actual container file and same serve HTTP folder directory that we had from before was actually mounted
05:22
in the home directory but we pulled the container, put it in that spot, I wanted the logs and give it the networking and then actually we're running on top of Debian so we just used systemd to launch it so we have the same command as before,
05:41
we have full path to geeks but geek system container, full path to the container itself, the same two shares, you need the path and give it the networking and then just go ahead and launch it. The whole thing runs under the BNW user so all the random files being uploaded
06:02
were also owned by it and as far as updating this every now and then, just ran geeks pull which pulled in new versions of, not of say graphvis or octave but new versions of PHP, new versions of nginx, all the other packages that I didn't have to go
06:24
and downgrade and then restarting the service, it just goes and rebuilds the container itself and it's there and available from the Debian, from the Debian host that this all was running on, we just went and said, point nginx at port,
06:41
what was it, it was 8880 and we got it from the outside so then, at a certain point in the future when it does go and crash and I have to go and rebuild the software, we have another command I haven't listed here, geeks list generation, geek system list generations,
07:03
that's system, you can list the generations from geeks pull, so it's geeks pull list generations, you go back to the previous generation when it did work, restart the container again and you're back up and running while you figure out just what changed this time.
07:21
So with that, we're hosting the old web app using new technology, newer technology, newer versions of everything and everything seems to be working.
07:44
Thank you again, we have time for one question.
08:04
Thank you. What's the outlook for you? How will you go about updating now that you have it on the known state so that you can move ahead, what are you planning? Right, okay, I'd have to repeat the question.
08:23
Okay, I guess I'll do it anyway just in case. So the question was how do we update the state? The actual package that we were given doesn't really get updated, it was more of making sure that there weren't anything left behind
08:41
from previous times we had restarted it. As far as the actual packages that we used to build everything, I had to downgrade GCC to GCC5, so that was one of the things that as we moved forward, I had to add that in to make sure that it worked again.
09:05
Yeah, I guess the rest of the dependency graph of everything that was running more or less because it's on top of GeeksGeeks keeps everything else up to date and then I'm just locking these versions in
09:21
at their old versions. All right, thank you Ephraim. That concludes our lightning talk, please applaud.