Rootless, Reproducible & Hermetic: Secure Container Build Showdown
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 44 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46134 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 20197 / 44
5
10
17
20
21
30
32
36
44
00:00
BuildingCore dumpComputer animation
00:32
Plane (geometry)Control flowAnalytic continuationFocus (optics)Game controllerInformation securityPlanningTable (information)BuildingSource codeComputer animationJSONXML
01:04
Computer-generated imageryInformation securityPairwise comparisonRootCategory of beingPhysical systemBuildingOcean currentRootPoint (geometry)Computer animation
01:39
RootProcess (computing)Vulnerability (computing)Basis <Mathematik>Operations researchPivot elementInformation securityComputer networkProcess (computing)RootGodPivot elementOrder (biology)Point (geometry)NamespaceCASE <Informatik>Principal ideal domainConfiguration spaceInformation securityRun time (program lifecycle phase)Kernel (computing)Boolean algebraDemonTerm (mathematics)Network topologyDefault (computer science)RoutingFlagComputer animation
03:46
RootProcess (computing)Run time (program lifecycle phase)NamespaceRootCodeComputer fileConnectivity (graph theory)Information securityGroup actionNamespaceBuildingChainModule (mathematics)Point cloudVulnerability (computing)SubsetSocial classProcess (computing)Cartesian coordinate systemLengthMedical imagingDemonGame controllerSpacetimeMultitier architectureIntegrated development environmentRhombusComputer animationLecture/Conference
05:34
HierarchyNamespaceRootKernel (computing)Computer fileKernel (computing)Multiplication signPoint (geometry)File systemDynamical systemLevel (video gaming)RootDifferent (Kate Ryan album)Bookmark (World Wide Web)CodeNamespaceInformation securityIntrusion detection systemTouch typingShift operatorData miningComputer animation
06:47
Computer virusCompilerFunction (mathematics)MalwareCodeInductive reasoningIntegrated development environmentCASE <Informatik>Compilation albumComputer virusCartesian coordinate systemRemote procedure callSoftware developerCompilerGame controllerSocial classBinary codeComputer animation
07:32
BuildingSource codeMathematicsIntegrated development environmentCategory of beingHash functionSelf-organizationDistribution (mathematics)Multiplication signMedical imagingCentralizer and normalizerFunction (mathematics)Reflection (mathematics)Projective planeIdentity managementBinary codeOpen sourceUniform resource locatorPhysical systemBitLevel (video gaming)SoftwareFormal verificationBuildingComputer animation
09:25
Library (computing)Identity managementBuildingSign (mathematics)Integrated development environmentPhysical systemSoftwareCodeChainProjective planeFormal verificationConnectivity (graph theory)Function (mathematics)MultiplicationProcess (computing)Independence (probability theory)Uniform resource locatorForm (programming)Event horizonMatching (graph theory)Software repositoryComputer animation
10:40
CodeCategory of beingLevel (video gaming)Electronic signatureIntegrated development environmentFunction (mathematics)Medical imagingComputer animation
11:12
BuildingSign (mathematics)Identical particlesFunction (mathematics)Hash functionFunction (mathematics)Group actionMedical imagingRevision controlLine (geometry)Local ringSoftwareCartesian coordinate systemSystem callOperating systemProduct (business)Validity (statistics)Personal identification numberBuildingCondition numberCache (computing)Identity managementMultiplication signProof theoryComputer fileSign (mathematics)Speech synthesisReflektor <Informatik>Similarity (geometry)BitContent (media)
12:36
ImplementationDistribution (mathematics)SynchronizationMedical imagingState of matterRevision controlHash functionSpacetimeSoftware bugEncryptionContent (media)Cache (computing)ImplementationDistribution (mathematics)Formal languageXML
13:25
Hermite polynomialsState of matterSakokuIndependence (probability theory)Category of beingLeakTerm (mathematics)BuildingShared memoryContext awarenessCycle (graph theory)Integrated development environmentProjective planeMultiplication signSimilarity (geometry)Boundary value problemMedical imagingForm (programming)
14:10
Computer-generated imageryOperator (mathematics)ChainDirection (geometry)Kernel (computing)System callBuildingMedical imagingFlagComputer fileSurfaceComputer animation
14:55
BuildingCodecState of matterValidity (statistics)ImplementationNamespaceSoftwareNetwork socketProcess (computing)RootKernel (computing)File systemMathematicsInternetworkingSocket-SchnittstelleComputer animation
16:07
BuildingComputer networkModel theoryDisintegrationBinary codeDemonState of matterPairwise comparisonBuildingRevision controlInformation securityCombinational logicModule (mathematics)Direction (geometry)MereologyProfil (magazine)Operator (mathematics)Goodness of fitNamespaceFlagDemonEndliche ModelltheorieFile systemRootContext awarenessKernel (computing)Latent heatProcess (computing)HoaxVector potentialSoftware bugEmulatorAdditionVirtualizationHacker (term)Parallel portPhysical systemMobile appSoftware development kitComputer fileMathematical optimizationCASE <Informatik>Computer animation
18:55
Group actionIntrusion detection systemNamespaceLibrary (computing)Binary codeCompilation albumComponent-based software engineeringScripting languageRootSystem programmingDemonComputer-generated imageryOpen setSoftware maintenanceEmulatorRecursionKernel (computing)System callAsynchronous Transfer ModeFront and back endsBuildingGoogolPoint cloudDefault (computer science)Similarity (geometry)Model theoryAddress spaceAndroid (robot)Java appletMaxima and minimaUsabilityCache (computing)Information securityInternetworkingCloud computingType theoryOperator (mathematics)Revision controlCASE <Informatik>Self-organizationInformation securityFront and back endsSoftwareAsynchronous Transfer ModeCache (computing)Service (economics)File systemAnnihilator (ring theory)RootBuildingGroup actionSign (mathematics)RoutingQuicksortMedical imagingHierarchyConnectivity (graph theory)Local ringGreatest elementMultiplication signLengthProjective planeDemonJava appletKernel (computing)Attribute grammarGoogolSoftware maintenanceSet (mathematics)Computer fileNumberExtension (kinesiology)Dependent and independent variablesDerivation (linguistics)Computer configurationInterface (computing)NamespaceMathematical optimizationDirectory serviceComplete metric spaceFunctional (mathematics)SpacetimeMechanism designBitExterior algebraOrder (biology)MathematicsSoftware development kitData conversionAdditionScripting languageFile formatOverhead (computing)Hermite polynomialsCompilation albumDifferent (Kate Ryan album)Run time (program lifecycle phase)WeightIntrusion detection systemKey (cryptography)Tap (transformer)Binary codeStreaming mediaEndliche ModelltheorieUsabilityPhysical systemComputer simulationChainComputer animation
26:28
BuildingMedical imagingInterface (computing)Service (economics)
27:09
Absolute valueLatent heatImplementationFunction (mathematics)Mathematical analysisBuildingOperating systemCategory of beingFunctional (mathematics)Direction (geometry)Cloud computingGame controllerShift operatorRevision controlInstallation artRight angleSelf-organizationLatent heatFunction (mathematics)Degree (graph theory)Software maintenanceMedical imagingChemical equationImplementationProjective planePhysical systemComputer animation
28:51
Interface (computing)Computer-generated imageryBuildingAbstractionGoogolPoint cloudWindows RegistrySource codeInterface (computing)Multiplication signBuildingPoint (geometry)Software repositoryComputer animation
29:25
Computer-generated imageryInclusion mapBuildingMalwareInternetworkingLevel (video gaming)Information securitySelf-organizationComputer animation
29:53
Computer-generated imageryContext awarenessProjective planeMultiplication signData centerBuildingNamespacePoint (geometry)Kernel (computing)Computer animation
30:38
Multiplication signMathematical analysisInformation securityDemo (music)Open setPoint cloudCodeHypermediaBasis <Mathematik>Pay televisionPlane (geometry)Density of statesLie groupMultiplication signFluid staticsMathematical analysisState of matterSoftware testingSet (mathematics)PlanningFunction (mathematics)Game controllerBeta functionConfiguration spaceSuite (music)Point cloudComputer animationXMLJSON
31:13
Projective planeCache (computing)1 (number)Level (video gaming)Data storage deviceBuildingShift operatorObject (grammar)Point cloudAdditionComputer virusAngleSpacetimeFile systemPhysical systemSupercomputerIntegrated development environmentGoogolSelf-organizationService (economics)Universe (mathematics)Twin primeCASE <Informatik>Internet service providerNumberFront and back endsExpert systemFreewarePerspective (visual)CodeBookmark (World Wide Web)Overlay-NetzClassical physicsPoint (geometry)Form (programming)Pairwise comparisonSoftware development kitPresentation of a groupKernel (computing)Software maintenanceRoutingMetropolitan area networkProof theoryXML
35:45
WebsiteSystem programmingExt functorComputer animation
Transcript: English(auto-generated)
00:06
My name is Andrew Martin and I am going to talk about OCI container builds. All the fun that we can have with them and it is quite a privilege to have many of the core contributors to some of the tools that I'll be talking about today present generally at the conference.
00:24
So I will mention a few names that I would then recommend you go and see those individuals talks. So, hello, I am Andy, a lover of all things breakable. You could say I'm a build fanatic and an advocate of continuous everything.
00:41
I'm a founder of control plane, which is continuous infrastructure and security practices with a focus on containerized deployments. And I want to talk about containerized builds and how to attack and defend them. So, we're going to talk about building stuff, building stuff that we can trust from source material
01:01
that we have no way of trusting from the table. We are going to talk about the properties of safe OCI build systems, ways that we can attack and defend them. A look at the current tooling and untrusted builds.
01:20
Is it ever safe to take arbitrary code and build it into a container? And with regards to containers in general, people have been speaking about this for years, hopefully it is well ingrained in your minds at this point. What is the heinous root of all evil? Unnecessarily running processes as root.
01:43
You wouldn't do it on your bare metal host, why do you do it inside a container? We can see here the wrath of the gods as they pass their judgment down to us. So, running as root is not vulnerable in terms of configuration in itself but it does provide a malicious pivot point if any other layer of a container's defenses are compromised.
02:08
How could they be compromised? Well, by the privilege flag, by sharing namespaces in the kernel and the reason privileged as a flag or as a boolean is so bad is because it turns off seccomp and app-armor
02:21
if they're enabled on the Docker daemon initially. Of course, they're not enabled by default on Kubernetes' use of runc. It busts out of most user namespaces and gives you a semi-faux PID namespace. It allows visibility and access to the host devices from inside the container.
02:41
At this point, I'm sure you can think of multiple ways to root the host from inside a privileged container. It is a very bad day for everybody. It also grants all capabilities. So, there is always a more nuanced and sensible way to give privilege to something than just using that flag.
03:01
We can see root was a prerequisite for the runc breakout earlier this year. And so, in order to generally enhance our security posture and defend against unknowns, zero days, et cetera, we want everything to be rootless, both inside the container at runtime for defense in depth
03:22
and to mitigate the ultimate lack of user namespaces, although, again, we will hear more from individuals this weekend, but also outside. Our container runtime should not be privileged in case of a container breakout, and then we have a process tree to follow to execute things as root on the host.
03:42
And if we are running our container runtime as root, the inmates are running the asylum. We're talking about docd, containerd, and runc, all enabling the execution of untrusted code, but running as the root user on the host.
04:04
So, rootless container builds. This means that the process building the OCI container image does not have access to a root-owned process or daemon on the host, or, ideally, any subset of root privileges. This is useful for building untrusted images,
04:22
for example, in cloud or PaaS environments, and also for defending against malicious supply chain components. Generally, we don't know exactly what our dependencies do at the application tier, and we certainly don't inspect our transitive dependencies unless we are using specific tooling to notify us of vulnerabilities there.
04:45
So, building without the root user protects us from a class of privilege escalation attacks. Of course, everything in Linux is a file, and discretionary access control to those files is specified by users and by groups.
05:03
So, running inside a container as root without a user namespace means that for any other namespace that is disabled, we are moving closer and closer to root on the host. Now, we're wrapped in security modules and various other layers, but we are focusing on defense and depth,
05:20
the many layers of the onion. Some build tools will create a user namespace to get around this problem. Others go to great lengths to avoid doing so. So, what is the big deal about user namespaces? They are the perennial Linux kernel security frontier and a difficult piece of code to write
05:42
because they spread across so many touchpoints in the kernel. In a user namespace, UIDs in the guest map to different UIDs on the host. That's ultimately the point. So, root in the user namespace has UID 0 and full capabilities, but obvious restrictions apply.
06:03
And again, my personal favorite of mine, that is, as a patch set almost shipped, the LXD guys are here. They've been working on ShiftFS. They've taken a patch set that was languishing for a few years and brought it closer and closer to fully productionized usage.
06:24
Again, I would urge you to go and see Christian Browner's talk tomorrow and examine the issue further with him. So, what that means, that is dynamic remapping of user IDs similar to user namespaces, but at the file system level.
06:41
A different approach with different compromises, but exciting times. So, we want reproducibility in our builds as well. This stops people tampering with our build artifacts. There have been real-life compiler attacks such as XcodeGhost,
07:02
an attack on the Xcode IDE that embedded malware in the compiled binaries and artifacts that were generated. This was essentially remote access control, or RATS, embedded in those binaries. And the Win32 induction virus alters the Delphi compiler's output,
07:22
for those of us who remember rapid application development with botnets enabling code in that case. This may sound familiar, and the fix for this class of attacks is reproducible builds. So, what are the properties of a reproducible build? A build is reproducible if, given the same source code,
07:43
the same build environment, and the same build instructions, any party can reproduce bit by bit identical copies of all specified artifacts. Let's start with operating systems. Compiling the same package in two different locations has, for many years, meant non-binary identical artifacts.
08:06
This build time non-determinism makes reflections on trusting trust difficult to disprove. We don't know the ultimate hash of the artifact that we expect, so how do we know nothing has changed between two builds? And, fundamentally, that change could be either the source code
08:23
or the build tooling that we used to build our artifacts could also have been attacked and compromised. The premise here is that if we don't trust our build tools, we can't trust their outputs. Projects like Debian have been moving towards fully reproducible builds for all their packages.
08:42
This creates an independently verifiable path from source to binary code. There's a lot of work going on at the distribution level, and we want to be able to extend this to container images and open source software that we pull from untrusted locations
09:00
such as GitHub, NPM, Maven Central, et al., and especially compiled artifacts because they are most difficult to introspect once they've been delivered into our organization. The ultimate goal here is to be able to run the same build in different locations and get the same output.
09:21
This is fundamentally a question of trust, and we should trust these men. From our build tools and environments, through our own software, and to the packages and libraries consumed in the build and runtime of the system, the trust goes all the way back to the identity of the user committing the code.
09:41
GPG sign your commits, of course. But what happens if, like Ken, we don't trust our build tools and supply chain? Well, we can rebuild the same package in multiple locations. If our builds are deterministic and reproducible,
10:00
we can hash and sign the outputs and then compare them to other build forms. Any non-matching builds are then subject to scrutiny. Attacking this system requires either a shared supply chain attack or individual assault on each node or farm, raising the difficulty bar significantly.
10:22
This is an independent verification of trust in the things that we're concerned with, the build tool and the artifacts that it produces. And this process significantly de-risks that component of our supply chain. Combined with tooling such as Intoto, which is the project that the Debian
10:42
Reproducible Build project utilized, this requires GPG signatures for each stage of a build. So we can run duplicate distributed builds and check the output signatures as the code is built to give us some certainty our code is not compromised
11:01
or that all our build environments are equally compromised. So to achieve this property of reproducibility for OCI image builds, we require local or pinned dependencies. If we don't pin our dependencies, then we are pulling the latest version
11:22
of whatever operating system package or application dependency our manifest requires. Non-deterministic, sorry, no non-deterministic network calls or non-determinism in general, perhaps easier said than done. And of course, an identical product every time.
11:42
No time-based behavior or outputs, identical output ordering, so maybe locale sorting is important there, and bit-for-bit similarity. Finally, signable and tamper-proof outputs. Of course, as the OCI image is addressed by SHA-256 of its contents,
12:03
we can easily see when any of these conditions are unmet in an individual OCI build. However, non-deterministic actions like network calls can't be identified in Dockerfile run commands and so require a user to add things like
12:22
checksum or hash validation for downloads in line that will fail a build on an incorrect hash and essentially trick the Docker image cache into being reproducible. Speaking of Docker images, again, another talk that we'll have tomorrow from Alexis Sarai on OCI v2.
12:42
The v1 spec is a tad ropy in places. It was derived from the state of Golang as of version 1.6, I think. I'm sure he can correct me. And includes some specific bugs from that version of the language, especially to do with the tar implementation.
13:00
OCI v2 is underway. It contains some nifty fixes, not only for reproducibility, but a proposal for rolling hashes for better image caching and content distribution. Watch that space. The talk is tomorrow morning, I believe. And there's also an interesting proposal from Brandon Lum for encrypted image layers
13:21
built into the OCI spec as well. So, on to the final property, hermitism, related to the practice of hermitage, seclusion, isolation, independence. In container terms, our builds should not impact each other, leak state, or indeed be knowable
13:41
by another build. And we shouldn't rely on things outside the build context. Hermitism also makes building images in multi-tenanted environments practical. This means that we can share build farms across projects with similar trust boundaries and reduce the cost of running build workers
14:02
and generally reducing CI cycle time. Okay, how do we attack a build? If the Dockerfile is untrusted, malicious commands in the run directive can attack the host. Or, despite a trusted Dockerfile,
14:20
a malicious or compromised image specified in the from directive has access to other build secrets. Or a malicious image has a nefarious on-build directive. Or, with Docker and Docker, because that requires the privilege flag for the nested Docker to perform host privilege operations,
14:41
we can then exploit the privilege chain to break out onto the host as we're running in a privileged container. And, of course, we can get out of the container in all the traditional ways by attacking the kernel syscall surface. So, to protect our builds, what do we do? We can prevent network or internet-bound egress.
15:04
Why is this needed anyway? It's better to pull our dependencies pre-build or from a local repository so that we know they won't change, and we can perform validation and scanning on those artifacts before we even pull them into our STLC. We can isolate ourselves from the host's kernel.
15:24
VMs are not really the essence of containerization, but nevertheless. Certainly, implementations we're now seeing are blurring all the way from gVisor to Firecracker with containerd shims all over the place. We can run run commands as a non-root user in the build's file system.
15:45
We don't want access to the Docker socket or to change global state in the container. And finally, we can run the build process as a non-root user or in a user namespace to prevent leakage of UID0 privileges from the host.
16:02
We should be in as many namespaces as possible. So, do you want to get burned? This is where we can get burned. I will leave these here for posterity and move on to a comparison of the existing state of build tooling.
16:25
Okay, Docker build version 2. Called buildkit and among a host of useful features including secrets mounting, it can run rootless. This protects the system from potential bugs of buildkit, containerd or runc.
16:42
That rootlessness uses rootlesskit, a kind of Linux native fake root emulator and everything, including buildkit, can run in a user namespace. This, and as we will see for many others, requires set UID flag set on apps.
17:05
This is integrated with Docker from 1806 and it can run as a standalone daemon. Rootless buildkit can be executed in Kubernetes courtesy of proc mount, which is an addition to security context. Again, more Jess Fras goodness.
17:21
What this does, it allows parts in proc not to be masked. Now, of course, there is still a compromise here because when we start container, of course, the container doesn't exist. It's not represented in the kernel by a struct or anything tangible. It is a combination of security modules
17:40
and namespaces and features and some nice UX, in Docker's case, that give us a container. One of the many fudges and clutches and hacks that are required is preventing access to some of the kernel virtual file system APIs. Masking proc is masking specific parts in proc
18:01
to prevent, for example, a process inside a container tampering with the host's kernel. So, unmasking proc should be considered operation based on your risk profile of your organization, as usual. So, why is this important?
18:22
Because buildkit uses proc to spin up nested containers. It is also likely, ultimately, to derive an entitlement model inspired by Moby entitlements. If anybody is involved with that, I would be interested to contribute and talk to you today. So, this will allow fine-grained permissions
18:42
around commands in run directives. And finally, buildkit has this optimized parallel step runner that is unique to OCI builders, as far as I'm aware. Good job, Docker. So, on to Jess Fraz's image.
19:00
This is a derivative of buildkit that re-implements a number of the buildkit interfaces to be unprivileged. It still requires setuid binaries, as does buildkit, and a lot of this work is just to get apt to run. What that essentially does is allow sets of subordinate user and group IDs,
19:21
which is the route around that many of these tools take. As you might expect, it goes all out with namespaces and setcomp, and as it's using a user namespace, it is fully unprivileged on the host. Nice. LXC, again, the team here, has an honorable mention
19:42
as it is the old faithful of container runtimes, powering the first versions of Docker and still going strong in Ubuntu. Despite using a different image format, it does actually support OCI containers via an OCI conversion script. If you use LXC with the scripts,
20:00
you can run containers fully rootless. This is not proliferated as much, but it's damn cool. UoC, again, this is an Alexa Sarai project, one of the original OCI manipulation tools, which takes a different approach to everything so far. Alexa is a runC and kernel contributor and maintainer,
20:24
and he uses some funky tricks to simulate rootfulness. The fact that he is a maintainer on those projects means that work required to get UoC to run has gone back upstream, and the commonality between a lot of this tooling
20:40
is that patches and changes have to be made further upstream, either runC or up in the kernel, to support new functionality required just to try and run unprivileged containers. It's transpired to take far longer and be much more difficult than we expected, but many thanks to the individuals doing the work.
21:01
UoC is rootless and doesn't require any special file systems to be rootless. It currently doesn't use any kernel name spacing. It's all VFS-based, and it should be noted that a lot of tooling in this space operates by recursively choning a directory to get around ownership issues.
21:22
This is an overhead, and UoC uses an extended file attribute to get around that particular timing issue or duration issue. It's a component of a wider build tool and maybe extended shortly to watch its own builder watch that space.
21:41
Kaneko is broadly a Google-sponsored project. It's used as the back end for, in fact, I say Knative build, but Tekton, I suppose, is the successor to that and targets a number of different build modes, notably Kubernetes, although, as you can see, GVisor is a particularly interesting runtime target.
22:04
It does not depend on the Docker daemon and starts a container using one of the noted modes. It does, however, use root inside the container. This is so it has permissions to unpack an image and run Dockerfile commands as the root user.
22:23
However, this opens it up to some of the issues mentioned previously. I will now ask people, now that I can go and target them directly, whether Kaneko was vulnerable to the RunC breakout that we saw at the beginning of this year. I'd be interested to get to the bottom of that one.
22:41
And there is an open issue to do with the hardening of the runtime against malicious from images, which is a little niche, but the fact that these sort of problems are being considered is probably a good sign about the general security strength of the build system.
23:01
Builder is Red Hat's answer to Docker build. It can run in various modes, with the rootless mode being preferable for our use case. Slurp for netns, another contribution from the rootless containers project, is an interesting addition. So this is more work from Alexa
23:21
and a Docker engineer called Akihiro Suda, who works on build kits for Docker. Between the two of them and some others, they are pioneering the rootless containers project, which is linked later on, and patching these things back upstream. Network operations are privileged.
23:42
And so in order to bypass the requirement for root to interact with these long-established kernel capabilities, Slurp creates a tap interface for network namespaces. It's a lot faster than the alternatives and is seeing general adoption as a mechanism for networking in the land of rootlessness.
24:01
See also user netties. Makkisu is Uber's response to this set of requirements. It is similar to Kaneko's approach in that it runs inside an unprivileged container somewhere, but it goes to great lengths to ensure distributed cacheability,
24:20
running Redis for distributed caching and providing local cache options too, including expiration and a fresh hashbang commit annotation. Another Google tool, Basel, I'm still not entirely sure, much hermetism, but at great cost. Run commands do not exist.
24:42
And there is much hoopla, consternation, wailing and gnashing of teeth online about this. But Basel does fulfill our base requirements admirably in that it can build anything, it can do so deterministically, but at the cost of usability, which is always high with Basel,
25:04
which is a great shame as it can build practically anything thrown at it, but you will notice that things like Istio, which started with a Basel native build toolchain, rapidly moved away. The adopter tax is quite high to learn.
25:25
So useful tool, great project, but perhaps not well suited to encouraging contributions. Google Cloud Build, this differentiates itself in a couple of ways. It is a managed service. It is pretty damn isolated and hermetic,
25:40
but probably not a usable build target for organizations with the type of security requirements that care about rootless container builds. For everybody else, this is a great option. And of course, it's reliant upon the cloud provider security model, which is not always fully disclosed, shared responsibility, etc.
26:02
Your mileage may vary. Jib is a bit like an opinionated Basel, very narrowly scoped to Java projects that need to build Docker images, but it has an optimized cache workflow that reduces build time. Is anybody using this here, incidentally?
26:22
Anybody? No, I haven't met anybody that uses it yet, but it's here for completeness. And again, do we have anybody building Nix? Yep, yep, we've got a few. So yep, Nix is the ultimate reproducible build tool, if you like. Google's Nixery offers Nix as a container build service,
26:44
which means by specifying on the image name, so the path of the image, if you like, the dependencies you would like installed, it will transparently build that for you as the container is being pulled, or immediately before.
27:01
So another managed service, albeit highly specific, but it creates very small images with an easy interface. So, do these fulfill our requirements? Everybody can achieve rootlessness today,
27:21
thanks to work shift over the last couple of years, especially in run C, but implementation-specific caveats abound. No builds are unreproducible and fallible by design, but output is a function of run directive behavior, and as such, only Basel actually achieves that
27:41
without diligent use of the run directive. To expound on that slightly, if your organization is able to allot the engineers to pin all of your package installations in an apt install, generally when packages are updated,
28:00
the older version of the package is removed. So your build will fail when the upstream operating system package maintainers bump a version. So the question then becomes, do I care more about the deterministic property of my images or shipping things quickly and applying controls later
28:20
to determine whether or not something is malicious, for example? Again, it's a balance. Anecdotally, I see organizations do both. I like pinning everything, but it's overhead, and for a personal project, it's probably deeply frustrating. And finally, everything has a varying degree of hermeticism.
28:42
Nothing is absolute, and we may well need VMs to achieve this full lockdown right now, which is essentially what a cloud provider will do for us. But why choose one? Akihiro has a container builder interface to abstract building. This is here for interest,
29:02
and there are some benchmarks available from a point in time earlier this year on that repo. So what do you think? And trusted builds should be great. They are almost ready. There are attacks against everything that's not container builder,
29:22
which is great, but doesn't work against on-prem builds. And in the wild, for the kind of ultimate isolation we're talking about, people are generally using VMs around fully untrusted builds. And by fully untrusted, we mean arbitrary, potentially malicious code pulled from the internet or potentially from a malicious insider
29:44
if we care about that level of security in our organization. I expect this to get close to full trustlessness in 2019, but again, I will verify that with better and smarter individuals over the course of this weekend.
30:03
So, Rootless Run C is already here. UserNet is Kubernetes' unprivileged in a user namespace, again, under the rootless-builds project from Akihiro and others. ShiftFS is with us at this point. Container versus hypervisor battle continues in data centers throughout the land,
30:23
but building containers in hypervisors seems a sensible compromise for mission-critical builds. And Heresy unikernels are still here, although they don't provide the most obvious container build context at this time. So, there are exciting times ahead.
30:42
In the last two minutes, I will just run through two short announcements. One is CubeSec is static analysis for Kubernetes resources. We finally rewrote it and open-sourced it. That basically gives you a JSON output determining the risk score of your stateful sets or deployment YAML.
31:03
Control plane are looking for beta testers for a suite of configuration and test tools for cloud-native infrastructure. Please hit me up if you're interested. And finally, thanks to all the people who did the work that I am describing. Thank you very much.
31:33
Any questions?
31:45
Sufficiently comprehend... Oh, hello. It seems pretty clear that not all of those tools can survive. And if you could bet on a racetrack,
32:03
which ones would be your favorites? From a hosted perspective, Google Cloud Build is excellent and fast and free. From a proliferation perspective, Canaco seems to be in the lead. A number of the other tools are either experimental or proof-of-concept stage.
32:21
Some are customized to the requirements of an organization. For example, the tool that Uber has is built to support a massive build farm, which is why they've added all these additional caching layers in. A note with Google Cloud Build is that they've now integrated the Canaco distributed cache. So of course there is a kind of Google centracism going on there
32:42
where they're trying to kind of standardize their own tooling and projects. They don't all have... I mean, for example, the work that Jess Frass did and Buildr are kind of spiritual twins almost.
33:00
So obviously BuildKit is probably the most proliferated one by virtue of being packaged and shipped with Docker. Canaco for building on a Kubernetes build farm and Google Cloud Build for the managed service angle. So in classic nerd fashion, mainly just two updates or tweaks.
33:24
I would also add to that thanks to Giuseppe Scavano because he actually contributed across several projects to get rootless builds working nicely. And Akahira Sudo, he's not a Docker engineer but a maintainer.
33:41
He works for NTT Japan. Ah, thank you. Yeah, I think I was aware of that but misspoke. Thanks. Couple of comments. One, ShiftFS has been rejected by the upstream kernel and they're trying to work around it by building it into the VFS layer.
34:04
Basically they don't want to have another file system layer on top. So ShiftFS is one of those projects that's always like two years down the road. The other one that Giuseppe's worked on is the thing called FuseOverlay which is basically a user space FuseOverlay file system.
34:22
He has some shifting built into it so that he can emulate a file system shift. So if you're going to talk about these, I think you should probably... FuseOverlay is really the thing that's powering a lot of the rootless, things like Podman and Builder and stuff like that is all based on top of FuseOverlay. Thank you. Is that a stage to reasonably do a performance comparison?
34:46
Yes, it's excellent. I did warn you that the experts were present. FuseOverlayFS is getting pretty decent adoption at this point.
35:02
It's even so stable that several universities have already adopted it and forked it for their HPC environments so that they can keep a main backend store or even their own object store and provide various academic use cases so that HPC has whatever the UID of the user that logged into the system.
35:22
It's fantastic. Wow, great. Thank you. Any more questions or I'm happy to ask you questions? Okay, thank you for your attention, everybody.