Passive filesystem verification
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 50 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43115 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 201823 / 50
1
4
11
12
13
14
16
17
19
23
24
25
29
30
32
34
35
39
40
41
43
44
50
00:00
SpacetimeContent (media)Data compressionFile systemJava appletSystem programmingComputer-generated imageryFingerprintData storage deviceObject (grammar)Dressing (medical)Identity managementHeat transferBounded variationFile systemNetwork topologyConsistencyMedical imagingMathematical optimizationAddress spaceData compressionNumberRead-only memorySign (mathematics)Content (media)Intrusion detection systemMultiplication signSpacetimeData conversionPhysical systemComputer-assisted translationWritingMereologyFraction (mathematics)Endliche ModelltheorieParameter (computer programming)File archiverFlagComputer fileControl flowLevel (video gaming)Fiber bundleDifferent (Kate Ryan album)Demo (music)Message passingBuildingPoint (geometry)SummierbarkeitInformationValidity (statistics)Library (computing)CASE <Informatik>1 (number)WordLink (knot theory)Function (mathematics)Slide ruleWindowType theoryStandard deviationImage resolutionVirtual machineMiniDiscSimilarity (geometry)File formatFlow separationService (economics)Decision theoryComputer animation
06:55
Data compressionSynchronizationInstallable File SystemFunction (mathematics)File systemFormal verificationDirectory serviceHierarchyFreewareSystem programmingLibrary (computing)Key (cryptography)Elektronische WahlMathematical optimizationAttribute grammarData managementEmailComputer architectureBackupKernel (computing)Extension (kinesiology)Function (mathematics)Computer fileRootkitDifferent (Kate Ryan album)File systemSubject indexingPoint (geometry)Single-precision floating-point formatWindows RegistryDrop (liquid)Multiplication signFile formatSource codeSpacetimeEquivalence relationReflection (mathematics)Default (computer science)ConsistencyFlagRing (mathematics)CloningMedical imagingSet (mathematics)Data compressionBinary fileDirectory serviceSynchronizationNetwork topologyEndliche ModelltheoriePoisson-KlammerMiniDiscInstallation artWordAndroid (robot)CASE <Informatik>NamespaceState of matterFile archiverSoftware testingLevel (video gaming)MereologyDatabaseFront and back endsCache (computing)iSCSIAngleReal numberHuffman codingComputer animation
13:44
SynchronizationSystem programmingNetwork topologyMenu (computing)HypercubeDirectory serviceLink (knot theory)File formatGroup actionSystem callPersonal digital assistantComa BerenicesComputer fileMultiplication signContent (media)Medical imagingSlide ruleCycle (graph theory)EmailNetwork topologySpacetimeLevel (video gaming)File systemGame controllerSystem administratorControl flowSpherical capHash functionTimestampGodBinary fileType theoryContext awarenessTelecommunicationLoop (music)CASE <Informatik>BitPhysical systemDirectory serviceArithmetic progressionDifferent (Kate Ryan album)SynchronizationPoint (geometry)File archiverEscape characterElectronic mailing listLengthPeer-to-peerLine (geometry)Endliche ModelltheorieElectronic signatureInverter (logic gate)Fiber bundleFunction (mathematics)Software repositoryFlow separationWordSystem callVideo gameMiniDiscComputer animation
20:33
System programming
Transcript: English(auto-generated)
00:07
In this talk, this is pulled from a past talk that I've done quite a while ago, so I'm not going to, I'm going to blaze through a lot of this.
00:21
You can go find the slides online later. There's links and or demos that you can see and try for yourself or have more information on. This is only a 15-minute talk, so there's more content here than 15 minutes. I'm going to blaze through that.
00:41
So yeah, passive file system validation. This is something that came out of an assumption for a lot of the container images. There are things that you can do to make file systems read-only, and it was always a rat hole of a conversation, because eventually somebody would say, well, you can mount it
01:04
read-only. Well, you could do this other thing. Or you might have a copy-on-write file system. Well, what if it corrupts? What if somebody does cap-dac override and they start going around you, and like, OS tree tries to solve certain things in read-only ways, and at a certain point you just have to make an assumption that it's either good enough for our use case or if there's
01:25
a break glass scenario that you can just wipe the file system and let the imagery pull itself or something like that. But how to verify that what you have on the file system is what you expect to have on the file system.
01:40
And so some folks would end up having the conversation of DM verity and these other things, but they're pretty tailored solutions. At the end of the day, you had to have some kind of a way to package, some way to have content addressability when you pushed and pulled those packages around, dealing with
02:04
compression, which is a pain in the ass, anyhow, all these different pieces, but still do it in the most generic, like the most common denominator approach. So I think that's all of them.
02:22
Anyhow, it's crazy when you deal with these packages and different packages try and solve this in different ways. Everybody in the world has seen like a sums file, you download some kind of content as a sums file and you have, you know, MD5 back in the day and then SHA-1, SHA-2,
02:41
SHA-256, SHA-512, and you have to have all these different IDs of something depending on what's most efficient for your machine to compute or what's irrelevant these days. And you know, different formats, we tried CPIO for a little while and it was cool, it's under a similar POSIX standard as tar archives are, but neither one of them accommodate
03:03
XAdders, and well, file capability is part of XAdders, anyhow, like long file names and just terrible stuff, they just keep recirculating terrible stuff. And even with container images, we're pushing around like 500 megabyte images in their
03:22
tar archives, several of them together, and sometimes it's so mind-bogglingly stupid, like you will make a, you will change the owner of a file and it will copy up, you know, a one gig file or something like that, like it's terrible.
03:42
And so now you have these bundles that are getting pushed around, they have to have some kind of a content addressability, like an ID that you're pushing and pulling them around, and for folks in here that have, who in here has used Docker either before
04:03
Docker 1.8 or Docker 1.10, really, Docker 1.8, folks, about half of you, who knows that the Docker ID that you had, it hasn't changed for a lot of people and now they call it digest, which I'm glad that kind of disambiguated that they're two different
04:21
things, but that the Docker ID from back in the day, it was a SHA-256 number, before Docker 1.8 was not content addressable. There was absolutely no part of pushing, pulling, building, having any hopes that the ID that you got is the same that's on disk, before Docker 1.8 was garbage.
04:40
And every single time you pushed and pulled the image, it mutated. It was actually a different tar archive every single time you pushed it. So having any kind of infrastructure for signing and assurance model was out the window.
05:01
And it's a, anyhow. And so even further, in this content addressability, is that once we started getting reproducible tar archives in the workflow so that we could have content addressability and build something like a digest, the things that some systems started choosing was the content
05:25
addressable address of the compressed archive, which was a, is a terrible decision. Like if you're ever in this space, have ideas, but they're terrible ideas, that always
05:42
take the content addressable ID of the uncompressed piece. Do yourself that service. Because in that situation it was relying on Golang at a particular compression level and the way that the Golang does the deflate and inflate Merkle, not Merkle, yeah, the
06:03
Huffman trees, Huffman trees, the cascading trees of the words. It was consistent to itself, but it was inconsistent with Gnu or some of the other ones, like there was two or three, Zlib, all three of them were inconsistent with each other.
06:24
And then often people would make certain types of like optimizations, like pass, especially if you were using Zlib, if you passed any of the flag parameters to do optimizations, then it might make your gzipped archive inconsistent with itself, from one compression to the next
06:42
it would be inconsistent. And that's not even to say like, oh, we've moved to XZ, bzip, LZ, LZMA, whatever, it's garbage now. So it's, there's some examples here where you can actually see, you have to do, even
07:00
for gzip from the command line, you have to do dash n, because in the gzip header it adds epoch, so one time to the next it would be different, but if you do dash n it doesn't include epoch, so you could have some consistency within itself. Real exciting stuff here. Zlib, this is using Ruby, but still, Huffman only, you have to add that flag, otherwise
07:26
it's inconsistent with itself every single time. Golang is terrible. Well even at Golang at some point, we were like, cool, it's consistent to itself, well
07:40
at some point they made optimizations for like architecture dependent optimizations for Go, and it broke all the compression, and so every Docker registry out there that had these one gig images, all of that cache was invalidated. Like, you'd go to pull an image, and it would fetch the new image, checksum it, and it got a different checksum, so all these registries were immediately having to like
08:04
rebuild all the images and reflush, like, oh my god, it was fantastic. So verifying at rest, regardless of like trying to make it efficient for whatever your backend is, whether you're r-syncing, files around, gosh, some folks were using
08:23
like iSCSI mounted over NBD to try and have like arrangements, but you still have to have this content addressable piece, as well as something like RPMQV, how many folks ever touched that?
08:41
It tries to show that what's on disk is different from the RPM that you originally installed, like has somebody tampered with some of the files, and so you can actually get some kind of an assurance model. Obviously if they're at that level anyways, they could have just modified the RPM database, whatever, but DM-Verity
09:01
DM-Verity ends up mostly being like an Android use case, but there was hope that it could mount and do these like read-only, like with absolute certainty that the file system is exactly at the state you expected, there's no corruption, there's no rootkit, whatever. There's been some IBM research
09:21
happening forever in the IMA EVM world, where you can have keys loaded into the key, you know, key cuddle of the internal key rings, potentially have like namespaces with their own keys, and within that namespace you can only see the things that your unshared space can have access to with
09:45
the keys that are loaded for it, and even if you escaped and got into another namespace that you couldn't see those files because the kernel wouldn't let you read them because they're not in your keychain, cool, we're not there yet, it's still happening. So a most passive, most generic approach at some
10:06
point when I was trying to figure out how could we do this, given all the file system approaches, given all the different package format styles, was something I remember from back in the BSD days, was mtree, and they used it largely just for like the file system layout piece of like them and their
10:24
source trees and ports. So it's in FreeBSD, OpenBSD, stuff like that, there is a mtree port that was done for Linux and it's kind of kept up to the
10:42
FreeBSD approach. Neither one of those accounts for extended attributes and some of the other, like some newer checksum, I think they've got to shop off 12 now, but xaddrs was the biggest thing, and even though FreeBSD has since added extaddrs into their file systems, they're not
11:01
using it heavily so they haven't added support for it in their mtree, those are some of the things that I wanted to tackle. So at some point I started, I set out and I write a lot in Golang, so I wrote a clone of, you know, a drop-in equivalent in Golang, but it added support for a few different
11:24
pieces like extended attributes. There are things available in the Linux space today, so if you dnf install bsd tar, you can actually say tar dash format mtree dash cf, you know, like you would make a file from this tree
11:46
and the output file won't be a tar archive, but it will be an mtree looking output. Linert has ca sync, which has an mtree output, so you can either do an mtree of a directory, which is similar to what go-mtree can
12:04
just do like an mtree output of a directory, or if you've made an index like a ca idx index, you can say give me the mtree of this backup that I did, it's kind of useful, and umochi that I mentioned a second ago that can like manage container file systems, when you do an umochi unpack, it actually
12:25
creates an mtree of that, of the expected output. So I've got like two minutes left. This is the go-mtree looking output. I don't think I included the xdatters in this example, but it's the new mtree output has this
12:47
notion of sets, so you can say in the following, until I unset or until each individual file overrides it, expect these to be the defaults, and so then it
13:02
will only have that keyword if it's different than the prior, and so then it walks through and it actually steps through directories, so those, these are children of the bin directory, and it's got a an annotation format to, like this
13:24
is actually angle bracket for the beginning part of test, the command that's, you know, square bracket, so it escapes that in its own way, and so you can see here that these these files, even though it's on disk, that this is
13:45
indicative that it has passed through a tar archive at some point in its life, because the nanoseconds have been trimmed off, tar can't support nanoseconds, but you can get a timestamp of those things, and then the SHA-256 digest or whatever. CA Sync does more of the old style mtree
14:06
approach, where every line has a fully qualified path rather than stepping down into the children. Some things are not quite as compliant with the old approach, it's not as much an issue anymore, people know how to handle some of these characters, but for the BSD world they considered use cases where
14:23
it might pass through a system that can't handle all the character sets for God knows reasons. Anyhow, but this is, you see this whole approach here, this file system also has passed through tar archives at some point. Leonard introduced this, and it jammed me up when I first tried it out, but SHA-512
14:46
digest was a was a keyword, but doing the SHA-512 trimmed to the 256 length was not a keyword, so I added support for that in go-mtree, but that's neat enough. This is just an example of doing scopio copy like I talked about
15:05
and my prior talk, Aaron Antonio talked about as well, doing an emoji unpack of that image that it just fetched into a directory called busybox bundle, and then seeing that there is now the the digest of that image dot mtree is
15:21
a file, and then you get that same output, so now you have something you can check. You can, even if the user mutated it, they can check that out against it later. I made go-mtree be able to reproduce and check against tar
15:40
archives, not just at rest file systems, so you could know what you're about to validate before it's like in flight. You might have a workflow where you get an archive, check the signature, lay it on disk, and then have something to check later for an assurance step. So there's several more examples here. You can go through them all, but libarchive or bsdtar.
16:06
So you could do that on most Linux systems, and it does the old-style approach, but it doesn't escape everything. So if you find yourself in any kind of an assurance model where this might be important to you,
16:21
these are the kind of artifacts that in some of the work that I'm doing, I'm including these so that you could have a sniff check of the contents of an image without having to download it and running an analyzer first. You could actually say like this is the sha-512 or 256 or 512-256 of the files that I
16:41
know are bad or whatever that you could actually sniff check an image like do a remote inspect and look for that image or get the file system layout, interpret the file system tree before you ever unpack a tar archive or whatever it might become one day like peer-to-peer. The work that I'm doing is
17:02
this repo. Obviously there's other things like ca-sync-entry and otherwise. I do encourage to stay in a communication loop so that these if you're either outputting or interpreting those manifests that they are compatible. Like I said I've added support and I didn't have a picture up
17:25
there but for extended attributes so that you could have se Linux context or whatever preserved on the files or otherwise because obviously if you set a file capability that makes a difference if you've added caps this admin or whatever to like user bin time that's a problem so that needs to
17:45
fail a check. Yeah and so then VBATS talks as these slides are up there as well you can see those examples and try them out later I don't do we have time for talks any questions so any questions from the audience have you
18:09
yeah it's on yeah have you ever looked at FS Verity FS Verity it's like the inverted you were on the file level really yeah it's pretty cool
18:21
it's like a work in progress from what's his name Ted is it Ted doing it no I have not basically I had only looked at the in Verity and ruled out that it's a narrow use case basically you write down the content of a file and then you do a Merkle tree hash on it and you add that at the end of the tree and then you call like and file control thing on this thing that seals
18:43
the file makes it read only and hides the Merkle tree from the user space is it how to like in like an X header no it's in the file so you write the entire file plus the Merkle tree and then you seal it and then it looks from
19:00
user space as if the Merkle tree isn't there but it verifies it every time you access the thing really it's pretty cool yeah I'll take a look at it's like work in progress I haven't tried it but sure it seems even I'd be interested if it could help and do some kind of attestation or if there'd be like a callback cycle there be that's cool cool no yeah Liz does that have
19:29
any relationship to the contents of the manifest in a OCI image spec this is completely independent but I very much think that it would be something
19:41
like an additional manifest or mime type descriptor that would just be shoved in so like you'd build the image and before you'd push it at a time where you could sign it that you could append some kind of artifact like this in it so that when you go to push it pull an image later that you would see it in the list of mime types and you could fetch the first piece like
20:04
what scopio remote inspect does you just get like the first bit of JSON and then you if your tool knows that that mime type like oh I know how to read that that you could then fetch that and get a better idea of the contents of the image and kind of yay or nay yourself on what you do next
20:20
that's that's that's a hope active hope all right I think we have a break next so thank you everybody