Cachix - binary cache(s) for everyone

Video thumbnail (Frame 0) Video thumbnail (Frame 3492) Video thumbnail (Frame 6924) Video thumbnail (Frame 8810) Video thumbnail (Frame 13047) Video thumbnail (Frame 14254) Video thumbnail (Frame 15497) Video thumbnail (Frame 16466) Video thumbnail (Frame 17553) Video thumbnail (Frame 18192) Video thumbnail (Frame 24997) Video thumbnail (Frame 26668) Video thumbnail (Frame 29437) Video thumbnail (Frame 30390) Video thumbnail (Frame 31018) Video thumbnail (Frame 32102) Video thumbnail (Frame 33522) Video thumbnail (Frame 34622) Video thumbnail (Frame 35440) Video thumbnail (Frame 37528)
Video in TIB AV-Portal: Cachix - binary cache(s) for everyone

Formal Metadata

Title
Cachix - binary cache(s) for everyone
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English

Content Metadata

Subject Area
Abstract
This talk is going to cover how to best use Cachix in hobby and professional manner to ease development and adoption of Nix. Talk is also going to cover the motivation how Cachix came to be, what are the design decision behind it and their consequences, usage statistics and where it's possibly headed. --- Bio: Domen Kozar: I've started using Nix in 2012 and after years of many contributions in free time, started using Nix exclusively in 2016 packaging up OpenStack and functional testing a cluster with scenarios. In 2016 I've founded Enlambda consulting which helped many clients like Snabb and IOHK to adopt Nix as a way to solve interesting complex software distribution problems in our industry. Robert Hensing: Nix user since early 2016, Nix and Haskell freelancer since 2017. You may have caught him on stack overflow with a solution, advice or bad news. He is convinced that the future is not just immutable, but functional. Link to slides: https://drive.google.com/open?id=0BwzQu6RqRR5zeFpBckw1M3Q3eXBPdW5kWmIyWmo0TTA0X0pr
Point (geometry) Scaling (geometry) Multiplication sign Control flow Multilateration Field (computer science) 2 (number) Product (business) Sign (mathematics) Roundness (object) Core dump Statement (computer science) Self-organization Video game Right angle Musical ensemble
Module (mathematics) Validity (statistics) Multiplication sign Projective plane Debugger Code Set (mathematics) Online help Cartesian coordinate system Proper map Graph coloring Performance appraisal Right angle Freeware Arithmetic progression
Point (geometry) Service (economics) State of matter Multiplication sign Virtual machine Data storage device Number Product (business) Cache (computing) Semiconductor memory Different (Kate Ryan album) Speicherbereinigung Energy level Mathematical optimization God Service (economics) Multiplication Key (cryptography) Software developer Projective plane Binary code Data storage device Bit Binary file Compiler Right angle Reverse engineering
Slide rule Randomization Email Cache (computing) Multiplication sign Binary file
Key (cryptography) Multiplication sign Software developer Binary code Client (computing) Limit (category theory) Binary file Cartesian coordinate system Sign (mathematics) Cache (computing) Self-organization Right angle Reading (process) Self-organization
Graphics tablet Backup Server (computing) Key (cryptography) Link (knot theory) Software developer Weight Web page Token ring Virtual machine Streaming media Web browser Binary file Public-key cryptography Cache (computing) Sign (mathematics) Cache (computing) Radio-frequency identification Self-organization Configuration space Backup
Server (computing) Service (economics) Virtual machine Mereology Casting (performing arts) Sign (mathematics) Goodness of fit Cache (computing) Internet forum Different (Kate Ryan album) Feasibility study Software testing Endliche Modelltheorie Beta function Binary code Feedback Data storage device Content (media) Bit Cartesian coordinate system Limit (category theory) Twitter Shareware Cache (computing) Video game Right angle Single sign-on Resultant
Source code Enterprise architecture Service (economics) Open source Disintegration Projective plane Stress (mechanics) Virtual machine Metadata Cartesian coordinate system Mereology Mixed reality Software Revision control Identical particles
Sign (mathematics) Cartesian coordinate system Shareware
Email Software bug INTEGRAL Web page Repository (publishing)
Installation art Software repository Code Repository (publishing) Web page Software repository Repository (publishing) Mass Summierbarkeit
Module (mathematics) Service (economics) Computer file Personal digital assistant Boom (sailing) Repository (publishing) Configuration space
Service (economics) Mathematics Trail Insertion loss Image resolution Object (grammar) Computer file Boom (sailing) Repository (publishing) Thread (computing)
Building Computer file State of matter Server (computing) Software developer Projective plane Attribute grammar Stack (abstract data type) Measurement Shareware Attribute grammar Shareware Performance appraisal Integrated development environment Mixed reality Revision control Repository (publishing) Gastropod shell Musical ensemble
Building Functional (mathematics) Electronic data processing Service (economics) Open source Multiplication sign Disintegration Control flow Mereology Shareware Formal language Revision control Computer hardware Repository (publishing) Gastropod shell Moving average Software testing Dependent and independent variables Email Information Server (computing) Planning Bit Binary file Cartesian coordinate system Performance appraisal Software Cube Mixed reality Revision control Right angle Gastropod shell Reverse engineering Spacetime
[Music] alright ok so the last talk before lunch break we've got Dolman and Roman has been busy be working on things that are going to make your life your life easier tooling and infrastructure things especially cash shakes and he's here to talk to us about that today so give him a warm round of applause all right hi everyone can you hear me well alright so yeah it's great to see such a big conference Nick's conf so I'm gonna start with the cheesy picture and a cheesy statement yeah this has been my passion to think about this question over and over again those that were here last year I had a talk how to running running Nick successfully for two years in production and during that time I've learned a lot and I think this is kind of the next step how do we make it scale in a way that you know multiple organizations can use that easily without hiring someone that that is you know in the core team of Nick's to be able to start using mix and and what are the gaps to get there and yeah I think I can personally break it out into two fields one is documentation there is there is a lot to be said on this topic this this talk is not going to be about that I will say that yeah everybody tries to do their best I also contributed a couple of I did a few experiments on the documentation topic but I think there is a second point and I started thinking about which one should come first and once we have the infrastructure I think the incentive to write documentation is going to come by itself more on that a bit later so I've decided to to spend more time on second point and get back to the first one once that's in a good sign
so I believe history what got me to this thinking and and what got us to to Kasich's and Hercules as Robert will present later I think that it's important to capture that so it starts in 2016 with Peter Simons the first person that kind of like put a documentation how to run your Hydra or CI fornix and yeah it's it's still there it's still valid it's there's a few things that change but if you go there you can set up your own Hydra and that was half year later I added this to to mix OS to nix packages Elco wasn't that happy about that because for a very valid reason heater was meant to be used for Nick's packages and it's very well crafted as this big build firm but it's not that well designed to be used in an organizational setting and those that might run one know the problems but I've said okay I'll help maintain this is the best we have so let's go with that right so it's there how many people use sudra all right so about 20 hands 30 maybe so yeah the first comment was now I'd finally have my own Hydra tried once had trouble didn't try again right so progress so in in about yeah few months later we figured out that Elka was right we need to as always right we need to we need to rethink this how do we how do we go forward so with Joe fish that's his nickname I don't I don't know I don't know how to pronounce his surname I'm sorry with Joe we started this project called hair color see I like them round idea was to rewrite basically to keep the SQL schema to keep everything except rewriting all the Pearl and the front end with an proper HTTP API and and I
created back then an elm application with a new front end and we did this is in free time and every time we had some you know free time after work or whatever we came back and we did some work after fixing Hydra we went back to reimplemented Hydra annum and then it
kind of died off for multiple reasons one is that it was a big project to do and second one we kind of needed to reverse-engineer Hydra and and we didn't really anticipate that that's that's not non-trivial so we try to save time on reimplemented because kima and asking some hard questions but we've wasted time on reverse engineering and also it was done in free time again so I started to ask myself okay how do we build something sustainable how do we build something that will you know not die again and what have we failed at this time and so yeah this is yeah this just took about a year or maybe less than a year to come to to a state that to a
state where it still wasn't working back then I was employed at i/o HK and I didn't have any time and then in April 2018 this year I left our HK and I said okay how do we fix this and looking around at different communities I found it the only way to do it is is to make it a product so that development can be sustained and that people can be assured that after some time it's still gonna be there and they can use it and it gets better and better and I don't okay it's just me right so what can I do I know that it's building us yeah at this point is hard and it's a lot of work even though it's like packaging right how hard can it be so this is just a different level so I said okay I drew the whole CI design and I said okay what's the most minimal thing we can do here and I said okay we can we can already have a binary cash as a service right you can register you get a binary Cash when you use it with your own CI your own development whatever you want right that's the idea so my first goal was yeah that setting up a binary cash shouldn't take more than a minute and you're good to go right whatever is you I use you plug it in and and and you start pushing binaries and you know your developers whoever can can go from there and use those binaries now step two is what's happening right now implementing garbage collection implementing different permissions multiple Reed keys multiple right keys and so on and that should be all easy right and I'm step three is it's finally document how to use cash sakes what are the you know what are the common things that people get into trouble when when starting to use necks and so on so we come back to to my to solving the first of the two points that I think are important so a bit of numbers it's running for almost 150 days it has more than 500 users registered it's using one point three terabytes of storage a lot half of users creates the binary cash and so far we had two hours of time time which turned out to be that some Haskell optimization actually in the compiler god yeah the machine down it used all the memory so this is still something I we want to pursue and fix but now at least if that happens again it should recover itself so hopefully that's that's fixed so yeah what I've been
working in the last couple of months is is private binary cash so for for people who don't want to publish their packages publicly and this is gonna be a paid feature random I'm gonna try to show that I have
a live demo and and a slide email so let's see which one works better how am i doing on time few more minutes and then alright so so now you'll be able to pick if you want your binary cash to be public or private the UI is not great this is just you know a way to show that
it works for now so you pick your your cash six subdomain and then you pick what is the team in your organization that will have access read access to two to the binary cash right the right axis is the signing key and I will also add support for multiple signing keys so who wants to have one for the CI and warfare developers or whatever that will be there so let's create an ex-con that was in an 18 and then yeah let's say you want to let's say that we want to give you know one team on github access whatever is in there I'll probably show something like members or something that's it's more clear what but who are you giving access to and then you create the binary cash
you get instructions how to install
annex how to install Kasich's you get instructions how to authenticate from your machine to tutor Kasich's and then you push the story pads one thing I want to make sure here the signing key is actually the secret key that you only get that is generated in the browser it's never sent to Kasich's and if you don't make a backup I don't have a backup for you so you'll have to create a new cache so I'll make sure this is this is more clear in the documentation and yeah that's basically how you push things with a signing key and then if
you go to to the link this is Mexican 2018 that caching start work well this is a development server so it's not actually there I you'll see that it's trusted by this github user that's the public key and then instructions how to use it so you can put you know inside your organization you can give link and again you install Nix Kasich's you authenticate because you need it's a private binary cash then you say cash excuse and that configures your Nix conf and your net RC configuration and you can use the binary cash basically so today my went well yeah if you go to
castings org there is a forum where you can sign up probably this weekend will will send out a private beta access for people to play with if you think this is cool and now Robert will talk about the second part of what we're working together the Hercules GI so yeah meanwhile though sorry before meanwhile he sets up you
can probably ask me a question or two [Applause] questions oh yeah over here so it's actually already relatively easy to set up private binary cashiers are three also so what's the main value add of cash X so the main value is that yes you can do that there are there's gonna be features that are gonna be on top of that Adele will make your life easier so like our bitch collection and you can you will be able to set up with the different single sign ons who has access to that and yeah you can use s3 for that but it has it's very limited on the other hand behind Kasich's there is an application server that which gives you possible which gives me possibility to do a lot of things like searching through through that through the binary cash and it's going to be connected to - Hercules so the bootstrap of of using cash is going to be faster right so it's yeah those are the the benefits in for now what's the cost model for Kasich's I do I didn't publish it yet and I I don't want to say what it's gonna be I'm still thinking about it most probably is gonna be storage oriented but yeah I'm interested too to come up with a way that it's it's feasible for people that it's not too limiting I always don't enjoy when when there is like an arbitrary limit and you just you know can use the service yeah still deciding on this but most probably everything looks like it's going to be storage storage size oriented but if you have some feedback about this I'm interested to hear it's it's not a no solution is ideal unfortunately or at least I couldn't come up with one maybe last question my epic watch is today going to tell me that I moved moved enough when I tried cash checks for the auto-updates to just to review them I found out that I need to add my own user into trusted users to use it although I'm only reading from the cache why is that yeah so it makes the binary cash can conserve you basically as Erica said in his talk right when you build the package the the actual result of the package can be arbitrary right the the the signing doesn't assure you that it doesn't sign the actual contents of a package right so if you are if if people would be able to just download things in your unique store from any user that's basically they could if I give you a you know non secure binding I mean adversary binary right if user guest one downloads can download from any binary cache then they can just put something in and then user you know guest you uses that package and it could be something not expected right so that's the the trust model UNIX wants to make sure that you there's on one machine cannot affect each other and that's why it's done this way you probably might want to have it with your hand it's probably gonna be too far away all right I guess I'll put it there for a demo all right so this is me this is what I look like and yeah I'm kind of somewhat kind of new to the next community so I I'm also on a bit active on stackoverflow by the way so if you asked a question there maybe you've received an answer for me I hope it was a good answer yeah so I'll tell a bit about myself some history yeah my story
started in 2016 when I was I had a neck stress on my on my desktop and yeah this was me deploying an enterprise scholar application on JBoss that didn't work very well but also in 2016 this was me deploying the enterprise Scylla application with Nick's on sense Wes actually yeah so since then I've I started freelancing and doing some some some research into how am I going to prevent these kind of situations improve things and obviously next was come to be part of that yeah so I added some side projects basically and yeah I started experimenting with an ex-gi and I figured out quite soon that building a
CI is not just a side project it really takes a lot of effort to build a good one so that summer I I talked to domin and we decided to work together and we redesigned ci design and this is what we came up with so we have Hercules CI and it's all reads of course it'll fetch sources from get up but they won't actually go to our service instead we'll have we'll provide you with an open-source agent that will fetch the sources build it you sky shakes to fetch dependencies use Kasich's to push them back when it's when it's done and report back to the Hercules CI service for notifications and a dashboard and so on so yeah these are like the distinguishing features for Hercules you will be able to control your own build firm so if you have some exotic things you need it should be easy to do so you can have as many machines as you want and yeah we strive to be the best and easiest CI for open source mix users and and for companies all right
so
here we go this is right this is our MVP and yeah so when you go to the to the application I just have to sign in with github will be
using our paired up integration to configure softlight permissions automatically
so now Hercules knows that I'm there and we can start configuring
so I'll install Hercules now on quickly CI itself we have a mono repo that has the actual code now it'll be set up so
that particular CI can can actually only access that single repository so now we can switch to Berkeley CI accounts and start configuring the agent
and it's just a little snippet of in this case mix-ups configuration we have a mix-ups module that helps to set this
up so all we have to do now is write that to agent up next file like this and deploy the agent all right
groceries showing up it's good so now
all we have to do is is making a change to trigger bills and I guess what I'll do instead is just push had to whatever
and you'll see it starts evaluating and now it's it should be building it's not finished yet the actual building is something that's almost done I mean as a feature so it'll be stuck in this state for a while but yeah we do have
evaluation working and so these are the attributes that are in in our project and you can see there's actually we found a problem by using Hercules it's its value yeah so we have a mix up as a as a development tool we're building it in our in our next file and yeah we linked it in the way that was suggested in the next obstacle mutation but we're not using next path - fine thanks packages that's actually a good measure - to make sure that you're building the right thing to make sure that some developers environment doesn't affect your project so yeah that that broke here turns out that looks like for documentation it depends on a path that's discovered - next path and it shouldn't so something will have to fix yeah so that concludes the demo [Music]
so you've seen this
right so obviously we're not done yet and we have some some further plans to make this a really great see I want to provide some kind of NIC shell functionality with Hydra for example you cannot do side affecting stuff at least not in a proper way and we want to to integrate this into Hercules side so we have a like a principled way to to perform side effecting stuff as part of your deployment pipeline yeah and we we're still looking into ways to integrate with container infrastructure like docker or kubernetes there's a huge design space we haven't really decided how how we want to do this and we would love to talk to you about about this and other ways you you are using NYX so we can provide the best experience yeah there's a private beta you can sign up for it we'll be sending out emails very soon to gather some more information and yeah if you if you sign up now you'll you'll be included and we love to it so thank you all right thank you very much two men and what and we have questions from the audience yes we do do my running thing again so what's your technological stack of which language this is written it and can we deploy it as well and the sub question did you manage to reverse engineer the Hydra cube we're on air so others can reuse it for their own CI attempts for example I micro CI and go right yeah so I stack is basically Haskell for the back-end services using um front-end obviously we're using NYX for packaging and deployments for now we're using NYX ops on AWS but we probably want to change that into something a bit more flexible yeah and so our 18 software is open source at least it will be when we start the private beta we will release the agent software and yeah it's open source so definitely have a look at it are we info are we invoke NYX so it wasn't entirely clear to me what the agents is for okay you said it again all right so that the agent that's the part that you're running on your own infrastructure and it's responsible for invoking Nicks for evaluation for invoking or basically mixed bills and yeah it'll fetch dependencies obviously and it'll upload those depend just built packages to a binary cash okay so the evaluation and building is not done by the Hercules CIE service yeah so at least at least for now like the hard work so to speak is it's all done on customer infrastructure we may implement a a more hosted version of the service in the future but right because that would be a very compelling feature yes yeah yeah so our first approach is to make it really easy to do it on your own infrastructure and probably in the future we will look into hosting the entire infrastructure yes for most people it's not required to to host their own infrastructure right because many applications are running on like commodity hardware so yeah for those people we want if we can make it even easier by hosting their agents we will do sir probably more questions nope doesn't seem so so yeah thank you very much again don't mind all that for your talk and so next up we've got a one-hour break for lunch lunch time enjoy
Feedback