We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

How secure is your build/server?

00:00

Formal Metadata

Title
How secure is your build/server?
Subtitle
a story of packages and trust
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
We have learned that we need to trust others, but as our parents used to say - don’t trust strangers. So we secure our production server more than ever. Yet, there is this no-man's land: “the build server”. We think it’s time to take a closer look at some of the good practices around securing builds & artifacts to improve our day to day level of trust. Please note that this talk replaces one entitled "Safe, gated and integrated GitOps for Kubernetes" that was due to have been given by Mohammed Naser, who unfortunately is now unable to present. We wish him a speedy recovery. Development has changed over the years, from doing everything yourself to a 3rd party package for every function. Operations has changed too, running your own servers is now considered an exception. To the cloud! We have learned that we need to trust others, but as our parents used to say - don’t trust strangers. So we secure our production server more than ever. Yet, in the middle sits this no-man's land: “the CI server”. We think it’s time to take a closer look at some of the good practices around securing builds & artifacts to improve our day to day level of trust. With Marked Sherman statement “Development is now assembly” in mind, the talk will focus more on the package/artifact/repository aspect. Less on the app security inside the code itself or at the OS/Machine level.
33
35
Thumbnail
23:38
52
Thumbnail
30:38
53
Thumbnail
16:18
65
71
Thumbnail
14:24
72
Thumbnail
18:02
75
Thumbnail
19:35
101
Thumbnail
12:59
106
123
Thumbnail
25:58
146
Thumbnail
47:36
157
Thumbnail
51:32
166
172
Thumbnail
22:49
182
Thumbnail
25:44
186
Thumbnail
40:18
190
195
225
Thumbnail
23:41
273
281
284
Thumbnail
09:08
285
289
Thumbnail
26:03
290
297
Thumbnail
19:29
328
Thumbnail
24:11
379
Thumbnail
20:10
385
Thumbnail
28:37
393
Thumbnail
09:10
430
438
Data compressionAlgorithmMereologySource codeFocus (optics)Product (business)Field (computer science)Computer animationLecture/ConferenceMeeting/Interview
Computer hardwareProxy serverInformation securityComputer hardwareData managementProxy serverPhysicalismServer (computing)Computer animationLecture/Conference
Key (cryptography)ThumbnailIdentity managementData modelFocus (optics)RankingGroup actionPhysical systemComputer networkDiscrete groupFormal verificationTheoryDynamical systemContext awarenessInformationGraph (mathematics)Theory of relativityData managementSemantics (computer science)Configuration spaceSystem programmingServer (computing)MathematicsAuthorizationTheoryIterationFormal verificationBackdoor (computing)Slide ruleWordComputer animationLecture/ConferenceMeeting/Interview
BootingPublic key certificatePasswordEncryptionMiniDiscPublic key certificateLaptopOpen sourceComputer hardwareAuthorizationOperating systemSoftware developerPoint (geometry)
Software developerHill differential equationSoftware developerMobile appLibrary (computing)InformationData storage device
ChainTransport Layer SecurityNormed vector spacePublic key certificateWebsiteServer (computing)System administratorSoftwareError messageFormal verificationForm (programming)ChainConfiguration spaceEncryptionPublic key certificateComputer animation
Radiology information systemSuite (music)EncryptionTransport Layer SecurityDefault (computer science)InfinityMobile appVariable (mathematics)Product (business)Integrated development environmentEncryption1 (number)Default (computer science)Computer animation
Electronic mailing listPublic key certificateMereologyFunction (mathematics)ChainInternetworkingDistribution (mathematics)Point (geometry)GoogolAngleEncryptionPlastikkarteDomain nameWeb browserView (database)Key (cryptography)Limit (category theory)Physical systemFrequencyRenewal theoryAutomationRSA (algorithm)Public key certificateLibrary (computing)EncryptionMultiplication signComputer animation
Software developerPublic key certificateGamma functionDomain nameFacebookPublic key certificateInterface (computing)InformationComputer animation
Row (database)EncryptionPublic key certificatePublic key certificatePhysical systemRow (database)Formal verificationDirect numerical simulationBitAuthorizationValidity (statistics)1 (number)Computer animation
Direct numerical simulationRight angle1 (number)Computer animation
Web browserGoogolVisual systemWebsiteAddress spaceGraphical user interfacePublic key certificateAdditionBinary codePartial derivativeInstallation artComputer configurationView (database)Scripting languageDirectory serviceState transition systemElectric currentElectronic signatureData integrityFormal verificationBinary fileGreatest elementBuildingVertex (graph theory)Server (computing)Goodness of fitOpen setBinary codeBitValidity (statistics)LaptopRight angleDirect numerical simulationComputer animation
Directory serviceState transition systemElectric currentElectronic signatureData integrityFormal verificationBinary fileScripting languageGreatest elementBuildingServer (computing)Link (knot theory)Radical (chemistry)File formatFiber bundleHash functionWeb pageRule of inferenceChainLaptopMultiplication signBinary codeStandard deviationElectronic signatureWebsiteMobile appFormal verificationData storage deviceSource codeComputer animation
Capability Maturity ModelRepository (publishing)Installation artScripting languageExpressionFree variables and bound variablesInformation securityVulnerability (computing)Multiplication signVulnerability (computing)Goodness of fitLibrary (computing)Information securityPort scannerCompilation albumVector spaceDampingRight angleTraffic reportingComputer animation
Scripting languageSurfaceCommon Language InfrastructureWindows RegistryModule (mathematics)Modul <Datentyp>Order (biology)Revision controlConfiguration spaceInstallation artScripting languageData managementComputer animation
Server (computing)Axiom of choiceCodeProgrammable read-only memoryInstallation artPoint cloudData managementPhysical system
Mathematical analysisOpen setPrototypeCodeSoftware bugLibrary (computing)Multiplication signPort scannerComputer animation
Binary codeProduct (business)Graph (mathematics)Coma BerenicesRegular graphView (database)Entire functionDatabaseCompilation albumExecution unitCommodore VIC-20Well-formed formulaPhysical systemElectronic signatureFormal verificationSoftware repositoryProxy serverLevel (video gaming)Game controllerLaptopState of matterProjective planeSoftware maintenanceMultiplication signComputer animation
InfinitySign (mathematics)Inclusion mapFile formatData integrityHash functionModule (mathematics)Cache (computing)Local ringString (computer science)Message passingFormal verificationAngleWindows RegistryIdentity managementSkewnessFlagRepository (publishing)Extension (kinesiology)Twin primeInformation securityCodeElectronic signaturePhysical systemInterface (computing)BitKey (cryptography)MultiplicationProjective planeProduct (business)Electronic signatureComputer animation
Server (computing)Enterprise architectureGroup actionHacker (term)FingerprintFile formatAuthenticationAvatar (2009 film)Point (geometry)Key (cryptography)Server (computing)1 (number)FingerprintComputer animation
Flow separationThresholding (image processing)Key (cryptography)Key (cryptography)Software frameworkMultiplicationPublic key certificateRootFlow separationMultiplication signDistribution (mathematics)Reading (process)SphereData storage deviceCartesian coordinate systemProcess (computing)Proxy serverSign (mathematics)Computer animation
Windows RegistrySign (mathematics)PiLocal ringRepository (publishing)Key (cryptography)RootRepository (publishing)Medical imagingKey (cryptography)Computer animation
Data integrityMenu (computing)INTEGRALWeb browserCodeFormal verificationKey (cryptography)Library (computing)Computer animation
CodeFunction (mathematics)CodeConfiguration spacePoint cloudFormal verificationComputer animation
Integrated development environmentBuildingSoftware developerComputerQueue (abstract data type)MalwareCodeBlogProjective planeCartesian coordinate systemLaptopSoftwareFlow separationSimilarity (geometry)Integrated development environmentMultiplication signVisualization (computer graphics)Quality of service
Server (computing)Direction (geometry)Software developerGraphics tabletServer (computing)Point (geometry)Computer animation
Trojanisches Pferd <Informatik>Statement (computer science)SoftwareGame controllerCodeSoftware repositoryComputer animation
Software testingSoftwareServer (computing)SubsetSoftware developerProjective planePhysical systemBinary codeComputer animationMeeting/Interview
Point cloudEmailComputer virusControl flowLimit (category theory)CodeIP addressMedical imagingPhysical systemVirtual machineLoginComputer animation
Limit (category theory)Key (cryptography)Gamma functionProfil (magazine)Key (cryptography)RandomizationComputer animation
BuildingPhysical systemCore dumpProcess (computing)DeterminismCodeSoftware developerElectronic signatureComputer-generated imageryPhysical systemResultantMultiplicationComputer animation
Source codeCodeInformation privacySoftware developerSoftwareFormal verificationBinary codeBuildingEmailFlagBinary codePoint (geometry)CompilerMultiplication signComputer animation
Binary codeIdentical particlesProgrammable read-only memoryLatent heatLinker (computing)Hill differential equationCompilerSet (mathematics)LaptopMultiplication sign
Right anglePhysical systemBootstrap aggregatingOperating systemCodeLink (knot theory)DampingComputer animation
Graph (mathematics)CompilerBinary fileMaxima and minimaBinary codeAbsolute valueSource codeLevel (video gaming)ComputerCodeNumberWeb pageFunction (mathematics)ComputerObservational studyAssembly languageIntrusion detection systemMassPredictabilityControl flowProcess (computing)Polymorphism (materials science)Entropie <Informationstheorie>Event horizonoutputLibrary (computing)CompilerAssembly languageFormal verificationBootstrap aggregatingSource codeHacker (term)Slide ruleFluid staticsRight angleWrapper (data mining)Binary codeComputer animationEngineering drawing
BuildingConvex hullPlastikkarteFormal languageSoftware testingParallel portSoftwareScalabilityCorrelation and dependenceMathematical analysisMathematical optimizationCompilation albumSet (mathematics)Medical imagingComputer animation
Computer-generated imageryPort scannerVector spaceGastropod shellRun time (program lifecycle phase)InformationDivisorBinary codeDirection (geometry)Electronic mailing listCodeComputer animationSource code
Computer forensicsPort scannerComputer-generated imageryReverse engineeringComputer fileMultiplicationMetadataBuildingReverse engineeringComputer forensicsVideoconferencingComputer animation
RootSoftwareSoftware frameworkChainFingerprintGraph (mathematics)Source codeTotal S.A.MetadataFlow separationProduct (business)Physical systemMedical imagingVulnerability (computing)Computer animationProgram flowchart
Right angleFormal verificationBitLibrary (computing)Computer animation
BuildingServer (computing)TelecommunicationGroup actionBuildingMultiplication signComputer animationLecture/Conference
Intrusion detection systemDatabaseServer (computing)CodeFormal verificationParameter (computer programming)Product (business)Asynchronous Transfer ModeBinary codeWebsiteBuildingVulnerability (computing)Source codePatch (Unix)Latent heatFerry CorstenRepository (publishing)Data managementPhysical systemReal numberComputer animation
Point cloudOpen sourceFacebook
Transcript: English(auto-generated)
For those of you like me who have like teenagers story of trust, I'll never clean up your room when they tell you And that's basically also a part of this talk Nerd alerts this was from a previous conference, but I guess here that's there's no alert everybody. Everybody's here
And I'd like to put the scope down We really only gonna focus on putting things from source to production So not running in production because that's another field of dealing with tampering and so on We're I'm not gonna tell you like scary stories about security because you know, that's all too easy. You all know that's important
I don't have to explain that to you I'm not talking about changing hardware because that's another vector, you know Nationwide attacks. I'm not going to go there as well I'm not talking about like human engineering or kind of tricking you into doing stuff that you don't want to do I'm not talking about physical bypassing access to your server rooms, you know
Trying to impress the talk as much as I can I'm not talking about key management. Although it's like important and here at FOS them. You probably want to manage the stuff as well Not talking about hardening your servers
Okay, so we're almost there what I'm gonna talk about tampering changing things unauthorized changes That's what I'm going to talk about I Leave this slide in here because promise theory for those who know it There's a lot of the word trust trust trust is in there But trust through verification and that's going to be the crux of the talk. Like how do we verify the stuff as well?
I don't have an answer for everything. So this is a research talk There's going to be a lot all over the place of things I found So I hope in next iterations of the talk. I will get better at Providing solutions as well. So but I went as far as I could from what I know right now in my research
So We're gonna start with you know, the laptop It's I know it's a curse to put like a Mac here and an open source conference, but that's a lot of the reality And we got a trust already the hardware vendor. We got a trust Apple. We got to trust the operating system So I'm gonna take that as a starting point
The developers laptop with the certificate author is on there, you know with everything in there and it's updated So I kind of start from there These Days a lot of the development is actually assembling more and more libraries, you know, if you're doing some development on a Mac
There's the App Store. There's Docker NPM if your node homebrew because you know Apple doesn't provide all the tools you actually need so, you know It's nice to have the community But we depend on a lot of other libraries actually to even start our job and get the information Before we can even start developing
So the first thing we trust is kind of all these kind of external dependencies and most of them are actually based on you know some form of TLS verification chain of trust And Well, you know, I don't have to pay to describe any configuration ciphers
You know all that stuff certificates that we think that is happening underneath because you know, we're using HTTPS and It should magically be taken care of right? Obviously, you know leaked or rogue CA certificates, but you know, that's another story. We'll leave that to the pros
And all the ciphers and so on if you So I'm taking this example of a node app trying to push that to production like the node By default has a lot of ciphers it supports so it has to support it because it's quite open So we're trusting it but we can get better at you know
Putting this down. It's it's as easy even as putting some environment variables and kind of saying these ones we don't want So it's a simple thing you can do if you don't Want just rely on node just doing stuff and there as well. I
Have to take my glasses on and off. I'm sorry What's interesting is that a lot of the we think about Certificates being validated. Yes, most of the time it actually validates whether the certificate has been expired But it doesn't look at revocations
So for example, you know you can Was it CRLs and so on you can verify it but in Several of the libraries it is just not available Let's encrypt s kind of made this better But we're still not there yet
If you want to go a step further, you know, you look at the SSL the certificate is valid you look at the revocation stuff The next step is actually like what other certificates got created for my domain Facebook provides like a nice interface To kind of check whatever
Certificate that was created in your domain. So that's a way of detecting walk certificates being created under your domain That you validate so you can put a webhook on there What if monitor your domain and kind of get information about things? certificates being created there as well
And if you want to go step one step further you can put in DNS The CAA record that actually says well if this record if the certificate has been created By which authority has it been created because maybe there's some dodgy kind of CA That creates a certificate for your domain, but here you can specify it in DNS. Which ones do you trust?
I'm not saying this is widely used but it's like one of the trust verification system we can use in the system So we we kind of you know Went a little bit deeper on HTTPS and validation than usual yes You know it's HTTPS and we're happy kind of a couple of things we could do
but obviously there's the next problem is kind of is DNS right because we want to do a request, but we want to do a DNS request and Well, and I'm sorry to say it still isn't solved and many would think like DNS sec is kind of the answer
But if you look at what a few of the major ones like NPM and github don't have the NSEC enabled So that was a surprise to me. I was like thinking yeah, this is the NSEC you know everybody should be doing it I know it's hard, but I would have expected these companies to be doing it
Even curl doesn't check it like there's like a six-year-old open pull request to do it like My trust in curl has all gone way down. I'm not saying it's not a good tool, but Obviously, it's people, but the more you look at the things under the hood the the scarier things kind of getting
There as well, and then you know okay. We got HTTPS out of the way We kind of trust that we got the DNS We kind of assume that trust as well, and then the next thing you do is like a homebrew says it's like You know run this kind of Ruby execute from github on your laptop
How many have done that? Yeah, scary, right So I don't know why we you know it's it's it doesn't seem to you know be able to get better to do it There's a tool I found like pipe this that allows you to actually view
Whatever has been piped to the bash command that you can review the commands being executed gives you some kind of visibility of the things that are happening But again like we just take that for granted right so homebrew right let's do it Node is a little bit better in a way that at least we can do the curl
Download and check the validity of the binaries They have GPG enabled so which is nice but Okay, just gonna go back on this slide. No, okay But it still is kind of a whole ceremony to go through right I rarely do it
But it's like one of those things It should almost get baked into the tool somehow in the future as a step that just says I downloaded it It's validated. It does all the checks currently across all the Package repositories, there's no standard way everybody does it differently and you kind of have to figure out and sometimes
It's not even well documented and you have to spend a lot of time to actually verify the stuff Doctor was an interesting one. I wanted to have Docker desktop on my laptop. I could not find it check some Somewhere on a website that was able to verify the DMG that I was downloading
The only way I could verify it was if I install it on my Mac Using app store and then I could verify the signature and that was documented But then again you we go through the chain of trust that we have to trust like Apple and and kind of go around that So that was a little bit surprising for me to see you know
We couldn't kind of do that with a docker binary and verify that as well Um Okay, I got like node I got like Docker a homebrew so now, you know, I'm gonna install some libraries, right
If you look at the libraries, how do you Think that a library is secure, right? Oh if there was like a million people using it. It's secure If it's like being there a long time it's secure So I haven't really found a good way
Obviously, it's kind of like verifying multiple vectors But there is no kind of good way to verify it There's a lot of vulnerability scanners that will report use CVS or something on something But if it's like a library not well know or a new thing. It's like really hard to verify
Unless you kind of download or compile everything from scratch yourself and go from there And PM install is interesting because it actually you know Like many of the package managers just runs also scripts when you do that So you can change a lot of stuff on your laptop as well. So you can for example
Use NPM install with ignore scripts and say well don't run it and get like a preview of the scripts It's going to execute and then do the execution. So it's like a next layer. You can verify What is actually going on underneath? It's it's very clunky. And of course, we don't want to have like, you know advertising in NPM who remembers this happening
Nobody so for a while when you did NPM install Instead of only giving you the packages that in install it will just show advertising in your CLI tool There was a lot of debates of something happening
But it kind of know shows you what these package managers could do to your system If you know, this is somebody who wants to do some stuff with it So you you kind of do the scanning you find all the bugs one of the problems obviously is kind of
all the dependencies and the libraries like a bug could be DDoSing it, but you're using it internally. So even with all the scanner tools It's really hard to figure out like what what do where do I spend my time patching? So That's you know, one of the the problems prioritization to do that
You can take a step further of the scanning of libraries and do it at a proxy level like JFrog x-ray allows you to do the scanning and not allow you when you download the packages through an Upstream repo and that they were like secure or verified or not. So you don't have to do it on your laptop Obviously you have to trust
JFrog x-ray, but it's kind of a another layer of control you can put there as well so this was just to point out that homebrew, the only thing it actually does is like looking at The checksum of the tar ball or something it downloads. So the actual recipe or was it the brew?
That itself is not signed And the maintainer said we're not doing that. We're an open-source project. We don't have time for that That was basically kind of the answer. So it's you know, it's kind of interesting On something we've used we're using so much on our laptops to install stuff that this is kind of the state where we are
You know, node is a little bit better they kind of signed to their Their dependencies as well. They have like checksums But you can see like this is all manual steps that you need to go through But at least you can verify it. You can get their GPG keys. You can verify the stuff. So that's nice
And obviously when you commit things yourself you can sign the signatures With your GPG key and push things to get but there's a I found an interesting project that allows you actually like multi comments or what is that multi submitters
That you're doing it that there is like a solution for multi signing it a commit so not only one person and then you can say well Before it goes into production multiple people have to sign that commit before it actually goes out. So that's another way of Verifying things there as well
You know if you're using github or something similar you when you push things these Fingerprints of the official host keys are actually documented so you can verify those as well Instead of just saying yes, I trust first trust and push it to the server as well It can kind of be like any anything that has like an SSH key at that point
You can kind of push to but at least here you can verify the official ones That brings me kind of to a principle of tough who has heard of tough a few people so it's kind of known in
More in the docker sphere is spreading towards more of the applications and the package distributions So it's it's tough stands for the update framework So they kind of made some principles and some guidelines of people pushing things to repos So the separation of duty is more like, you know one
Multiple keys and not one key to kind of push things thresholds much like the git commit signing they do the similar thing like multiple people Depending on the step have to kind of say yes, not only one So that builds like the consensus
much like Was it The certificate thing I talked about The revocation of keys is really important. So it's built into the framework that they can fast rotate keys To make sure that there's like no delay Offline keys. So the root keys don't need to be active all the time
So they can be on offline storage and then the trust goes up for more like shorter keys to be there and no need for the key sharing so Everybody can kind of delegate a key to somebody else depending on what the job they do So there's a principle look it up the the tough the update framework You know, I'm just rushing over here, but it's an interesting read as a concept
And this is what actually is used by Docker notary when you want to have push images in You know that are signed and secure those principles are used by Docker notary So you create a key you sign it very much like any of the package repositories as well
And then you can expect Whether it's signed with or correct to like you would do with a git commit but now with the docker image as well this actually has saved a major breach on the docker hub by Requiring multiple people to do a certain steps one of the keys got compromised, but they recovered in a way because they had this
principle available in in the setup We've talked about you know Node code or any of you know, Ruby or whatever code we talked about Docker images, but there is also something you can do for
Verification For for example for JavaScript libraries in the browser So you can build the checksum and you can make sure that there is an integrity check When the code gets executed in the browser as well so it's the checksumming and so on we just keep going and
We can verify more and more. So that's a good thing I'm taking it even further. We talked about code But no, this is probably the simplest Way in the days when I was running Apache like minus T Was my friends to validate the code or the config as well I'm not saying this is the trust but at least it gives you some kind of idea of what is happening
Happening and if you want to take that further now with something like tfsec who can basically check your terraform code To validate if there's something happening that shouldn't be happening as well. I Think that's getting powerful more in cloud configs that we can actually do that verification as well
Taking it one step further, so I'm just going up and up so What if I don't trust my laptop every time I do some stuff? I think Jess Rosalie has coined this idea of running every application inside of a docker container on the desktop
2015 this was a blog post now There's been a couple, you know visual studio code you develop code inside your containers and then we get better at Exporting like debugging it from remote you know
fedora has a similar project and Something like QoS goes a step further. They even give like every Application the way I understand it. They give every application their own network stack their own everything And Everything stays like separate ephemeral and it's actually used in
For journalists to make sure that when they're putting things in the news that they have like a safe environment Why where they can edit stuff and they can kind of make sure that it hasn't been tampered with So one of the point is that I know a lot of people say, you know servers kettles not pads But I think we're going the direction that it's gonna be the same for the desktops
I have when I said that I heard like developers screaming like I I'm I cannot have my own autonomy and so on I was like, yeah, I've heard that before like 10 years ago on the server world Giving pets names. We were just discussing that like Pokemon names or whatever
So we got the code verified we're pushing it now hopefully to the CI or We pushed it to the git repo and now hopefully the CI kicks in but what's interesting is that This often is outside our control. So it's like who watches the watchers who trusts the
People we trust so it's it's this is a nice paper to read if you want to know more I Don't know if anybody knows that project But Yeah, it's just you know It's in a fun way shows like anything can happen in your CI system that you you don't know that's gonna happen as well
And that makes you think like if you use like a saw solution How do we get that same trust? We can't verify the binaries. We can't do everything We pay them or we use them and we just assume The secure so that's like a big leap up
And I haven't got any You know perfect solution for it, but there's obviously they get better at exposing what they do So they show you the images there on your builds on they share that officially But we don't have any idea where they are running this image. So it's it's still like it's better
So I have a solution to run things on-prem so they would do the same orchestration But they run it on your machines So at least you can kind of get audit logs kind of see what commands get executed. So it's not perfect
some would have like More of They would officially announce that there was an audit of the code. So that's kind of how More and more they are building trust Some would allow you to limit the IP addresses Let's say if your build system is remote
They have you push things to your environment that at least you only allow the IP addresses of the people who are Pushing something to you and this is a nice, you know little trick for example, if you use any of the AWS CLI tools you can change the user agent and you can have like an EIM profile
That says like only this user agent is allowed so you can turn that into an almost random Key by obscurity that allows you to kind of say Well, nobody else even if though they have like the AWS credentials can actually use that if they don't know
The the random key that you put here. I'm not saying this is secure, but it's like one step more The people from The Bitcoin system they went a step further so they just went with what if we have multiple people compile it and then verify if this
Gives us the same result and then if most of the people get the same result We assume that is the best result and then we put the checksum in So not only one checksum because your tooling might be infected and just changing it there as well So I found out a nice way we we sometimes think about multi clouds. What about multiple CI?
What about using two CI systems and then do like verification and see both kind of created the same thing Again, it's like thinking I'm not saying this is actually being done, but it gives you some inspiration But this brings us to the hard point of doing reproducible builds, which isn't really that easy because
let's say you have like a Something like a Time in your banner or your header who really whatever and you're kind of recompiling it the next time you compile it You know, the binary will be different or the flags from the compiler or anything there. So
Deviant has spent a lot of time on making packages and builds reproducible Node, if I run it twice on my same laptop with the same compiler in the same settings The binary or whatever it builds Will not give me the same checksum
So it's something that I found interesting to see We would think like a checksum. This is kind of what we do One more step further What about the operating system itself? It turns out that this was from a Fosdam talk
To build a Linux nowadays you have to have a Linux right so somewhere we lost the link from being completely from scratch to build the Linux system Sometimes you see like bootstrap one, two, three when you do like GCC compiling like some of the bootstrapping is binary only
So people are still working on making that code as well So it's it's sometimes hard to verify to do that if you want to have a look at this This is I think there was a talk earlier today on that as well Like all the dependencies of the libraries. It's not that easy as it looks
And even like the bootstrapping assembler That you know people are working on that just to have like a from source verifiable compiler to make things happen I'm gonna skip that slide that slide right so we kind of said well make it repeatable make it repeatable
And then the hackers say fantastic now I can detect every binary because if I look at the checksum I know what binary it is so then this company said well You know what we're gonna build a wrapper that randomizes the checksums again To run it in production, and then we map it on the static
Checksums again, so it's kind of you know we're going around and around, but you know we're getting better The checksum is one thing I talked about Thinking about all the dependencies I think this is where Google Bazel comes in that they instead of saying reproducible builds
Which built the same binary they're working on The concept of hermetic builds were all the dependencies that you need in the build for your docker image or whatever They are specified. It is not just app get update or something they Kind of try to do that as well and not only for your dependencies
But all your also for your compiling toolchain so Bazel will download a known set of Compilers so that everybody in your team will use the same compilers and do the same installation as well And that brings us you know the attack factors quite often the discussion is on containers like hello
There's so much in the container and then so much vulnerabilities, and they're actually going the direction What if we only need like one binary in the container so distroless and that you know makes it a lot More difficult to do any attack factor there This Is what some people are working on so we when we know all the dependencies
We kind of have like a bill of material You know like the ingredient list this code has been built with this dependencies and so on Well there's many tools that allow you to reverse engineer docker files Because it's sometimes not that easy, and you don't know what's inside
You can take multiple approaches either from forensics or reverse engineering They're all in the notes you can see that later so some interesting stuff, but you know It's stupid that we throw away all the meta information while building and then Have to reverse engineer it again. I in my past company. I worked on
video stuff, and it's almost like When you have a DVD I know that's getting old with subtitles You know and you have to reverse engineer it from the image, but it will that's actually stupid
In total is a new Not new, but it goes one step further They actually not only specify the dependency sources But also the steps that they took while doing the build so they would sign every step of the build So you can verify what happened during the build as well
Then graphy us in the docker X system can use all that metadata to decide whether something should run in production or not Based on what has been in the image when a new vulnerability comes out They can kind of say well you know this is kind of something that this has been built now we know there's a
Severity one so you have to do about it, but that's something that can be integrated there as well But obviously you know it never ends, and we just keep going There as well so I'm getting toward the ends of my talk. I know all the tools that I've shown you are so hard to use right and
Sometimes I think about it We fought fought so hard to get the right to vote, but then you would say well I don't want to vote right and it's a little bit like this We fought so hard about getting the freedom for you know Using sharing libraries and so on and I think we kind of have to get better at the verification step because you know
That's one of the duties. We have as well to make our ecosystem more secure in there And I Think a lot of it has to do with the transparency Transparency building consensus so not like one
Group saying yes or no, but work on there as well So I think that was basically my thought. I don't know if you had time for questions Okay, anyone
Yeah, okay Mm-hmm I know there's quite some debate whether it's useful or not. Yeah, but I think If you automate it That's one of the problems Because if it's if it's just like I will do the from what I understand that the argument is about
when you automate the step then it's no use because it's There is no real verification happening, I don't know if if you have another idea on on there To build something
Yeah, but sometimes you have to I guess Okay, I think the question was is there
Yeah, okay. I'm gonna repeat the second one. So that that can prevent A build from happening if some the the the committee hasn't been signed I think that is probably looking at the verification and then putting an exit code. I don't know I haven't come across any specific tool in there. So
Yeah, okay, so that's another talk like I said, yeah
It is more dangerous but I think It's it's another, you know, it's another aspect of looking at things. I I think here I want to go in the prevention mode and not in the you know Even if you go into prevention mode you sure it's quite fast. Yes
Yes, yeah But the visibility of knowing what is running production
And I'm not talking about intrusion detection or logging or auditing but if you know what is happening and you can at least know check the Vulnerability databases and get some of the tools that do that you get you are better informed Whether you should kind of know patch the zero-day or kind of as fast as possible I'm not saying you don't have to protect your servers anymore
But at least the visibility is it's what I try to show here if you verify and get things visible At least you can see it. If you don't know it you just yeah live in happiness So that might be good as well. So One more yeah, yeah, that is what I think most of the distro the package managers are using
It depends it could it could not right if you're saying
if If I have like an S3 repository and my binaries here and my checksum is here You know on the same bucket and I have access, you know, yes, I can alter both So that's not really a solution But if I have the checksums on a website something else on a different system and I have my banners here
Yes, it makes sense Okay No, but it depends what you define a same source. Is it the same company? Is it another company? Is it the same website same server? So it's kind of depending on You know the more the more people are seeing it the the the better he gets there. Yes
Okay, I'm sorry find the other one