BCOS Monero Village - BCOS Keynote

Video thumbnail (Frame 0) Video thumbnail (Frame 12038) Video thumbnail (Frame 15805) Video thumbnail (Frame 22523) Video thumbnail (Frame 37246) Video thumbnail (Frame 49921) Video thumbnail (Frame 62342) Video thumbnail (Frame 74762) Video thumbnail (Frame 80768) Video thumbnail (Frame 92738)
Video in TIB AV-Portal: BCOS Monero Village - BCOS Keynote

Formal Metadata

Title
BCOS Monero Village - BCOS Keynote
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English

Content Metadata

Subject Area
Point (geometry) NP-hard Group action Context awareness Building Server (computing) Direction (geometry) Multiplication sign View (database) 1 (number) Insertion loss Water vapor Mass Computer programming Number Power (physics) Neuroinformatik Goodness of fit Causality Root Hypermedia Term (mathematics) Different (Kate Ryan album) Energy level Information security Area Curve Graph (mathematics) Information Military base Electronic mailing list Plastikkarte Bit Total S.A. Line (geometry) Cryptography Lace Entire function Maize Integrated development environment Order (biology) Self-organization Right angle Quicksort Whiteboard Communications protocol Resultant Spacetime
NP-hard Point (geometry) Slide rule Server (computing) Context awareness Open source Multiplication sign Direction (geometry) 1 (number) Mereology Event horizon Computer programming Causality Term (mathematics) Different (Kate Ryan album) Hypermedia Analogy Cuboid Energy level Software framework Endliche Modelltheorie Extension (kinesiology) Information security Metropolitan area network Physical system Social class Computer architecture Area Inheritance (object-oriented programming) Digitizing Database transaction Bit Basis <Mathematik> Entire function Type theory Cache (computing) Process (computing) Online service provider Self-organization Website Right angle Quicksort HTTP cookie Annihilator (ring theory) Row (database) Spacetime
Logical constant Group action Presentation of a group Code Multiplication sign Source code Set (mathematics) Insertion loss Open set Mereology Side channel attack Perspective (visual) Computer programming Front and back ends Web service Semiconductor memory Single-precision floating-point format File system Encryption Endliche Modelltheorie Information security Logic gate Descriptive statistics Physical system Software engineering Software developer Structural load Bit Virtualization Connected space Data management Process (computing) Order (biology) Right angle Quicksort Reading (process) Point (geometry) Slide rule Trail Dataflow Server (computing) Mobile app Service (economics) Computer file Patch (Unix) Virtual machine Heat transfer Rule of inference Number Wave packet Revision control Frequency Crash (computing) Causality Term (mathematics) Average Data structure Focus (optics) Wechselseitige Information Graph (mathematics) Inheritance (object-oriented programming) Military base Line (geometry) Cryptography Exploit (computer security) Kernel (computing) Integrated development environment Software Personal digital assistant Blog Point cloud Pressure Library (computing)
NP-hard Randomization Code Multiplication sign Design by contract Stack (abstract data type) Mereology Computer programming Formal language Fluid statics Mathematics Core dump Software framework Information security Vulnerability (computing) Physical system File format Block (periodic table) Software developer Feedback Bit Unit testing Price index Entire function Shooting method Process (computing) Chain Self-organization Right angle Moving average Whiteboard Quicksort Resultant Writing Server (computing) Mobile app Functional (mathematics) Statistics Service (economics) Open source Divisor Streaming media Rule of inference Touch typing Energy level Multiplication Host Identity Protocol Projective plane Mathematical analysis Interactive television Line (geometry) Incidence algebra Cartesian coordinate system Cryptography Personal digital assistant
Group action Greatest element Building Multiplication sign Insertion loss Usability Subset Mathematics Different (Kate Ryan album) Analogy Core dump Flag Circle Extension (kinesiology) Determinant Information security Physical system Identity management Area Texture mapping Regulator gene Data storage device Electronic mailing list Stress (mechanics) Database transaction Instance (computer science) Public-key cryptography Electronic signature Hand fan Process (computing) Self-organization Right angle Figurate number Quicksort Cycle (graph theory) Whiteboard Resultant Computer forensics Spacetime Geometry Point (geometry) Laptop Server (computing) Game controller Service (economics) Open source Online help Login Metadata Number Product (business) 2 (number) Revision control Centralizer and normalizer Average Internetworking Energy level Backdoor (computing) Mobile Web Pairwise comparison Dependent and independent variables Standard deviation Shift operator Information Uniqueness quantification Physical law Projective plane Interactive television Mathematical analysis Plastikkarte Incidence algebra Cryptography Limit (category theory) Peer-to-peer Uniform resource locator Maize Integrated development environment Software Personal digital assistant Password Game theory Abstraction
NP-hard Group action Game controller Service (economics) Source code Open set Information privacy Rule of inference Workload Latent heat Strategy game Different (Kate Ryan album) Term (mathematics) Operator (mathematics) Cuboid Energy level Flag Software framework Endliche Modelltheorie Information security Position operator Standard deviation Regulator gene Database transaction Bit Cryptography Process (computing) Software Integrated development environment Normal (geometry) Right angle Pattern language Quicksort Spacetime
Sensitivity analysis Group action Code Multiplication sign Source code Mereology Information privacy Computer programming Strategy game Bit rate Different (Kate Ryan album) Software framework Arrow of time Information security Vulnerability (computing) Physical system Area Regulator gene Digitizing Shared memory Sound effect Bit Instance (computer science) Type theory Chain Right angle Quicksort Arithmetic progression Pole (complex analysis) Point (geometry) Slide rule Dataflow Mobile app Service (economics) Open source Connectivity (graph theory) Similarity (geometry) Code Workload Frequency Term (mathematics) Operator (mathematics) Software testing Backdoor (computing) Computing platform Information Mathematical analysis Line (geometry) System call Performance appraisal Maize Integrated development environment Software
welcome everybody welcome Philip and I hope you enjoy the talk after the talk if you have any questions that's when everyone is kind of available and answer anything you have Manero elated or coinbase related or cryptocurrency related I guess as well and blockchain related right they're all kind of related in some way okay so enjoy the talk and hopefully talk to you soon Hey look at that computer worked I'm gonna apologize in advance for my voice it's it's not keeping up super well this weekend but with their work with the amplification hopefully it'll go okay along with some water as well good morning kind of good group hecklers here today that's gonna be a good time thanks for coming out to the clock good morning I'm sure some of you just stayed awake until the decock in the morning so more power to you you can go to bed soon as I said my name's Phillip for the last two and a half years I've led security at coinbase normally this is the point where I ask the question who's heard of coinbase but um I'm pretty sure that's an obvious question here how are you guys our customers nice that's a half-ish really cool thank you so I'll skip the the the sort of what is clean based but so the other way I describe what I what we do is we are on the world's largest CTF with the with the highest stakes sort of outcome all right so we store somewhere on the order of ten to fifteen billion dollars of cryptocurrency which is not something that anyone wants to lose right before we get it all that stuff I'll just want to part about the elephant in the room for a second I'm the first talk of the day half veer asleep half of you're drunk from the last night but so I appreciate you came out and I think what I want to sort of give you back for your attention here is a bit of insight into what it takes to protect a modern crypto currency exchange I don't know why that's hard in my point of view as well as some sort of view points into into where I think we as an industry you need to get better and need to get better in order to push crypto actually push crypto to the masses not just you know a very small boardroom or very small room in the middle of a DEFCON so I personally I've been doing this gear thing for a while I try to do I work with a very simple lens it says where can I find the most interesting attackers and and that let me do a bunch of different laces over my career but I will say crypto currency is I think takes the cake so far in terms of interesting challenges and really in terms of doing things that no one's ever done before and in this context there's been pretty cool I would be remiss if I didn't say coinbase is hiring economist is always hiring for security specifically but really across the board if what I'm talking about like is interesting to you you want to learn more come grabbing happy to chat so bottom line what are we gonna talk about today number one we're talking about a little bit the industry at large give some context on why cryptocurrency is actually a hard or being an exchange is actually a pretty hard problem we're gonna talk about a bit about how clan-based looks at the problem I'm not gonna go through our corn bases entire supergate program because we'd be here way longer than an hour but I'm gonna head on a few sort of interest areas that I think you're interesting and how we execute security against in this environment and then we're gonna talk a little bit about the industry as a whole and and where I think we can improve and do better and actually push ourselves out there and become the kind of industry that my mom is okay investing in it's really my bar when I think about this stuff cuz I think about my mom right if she would if she could interact with with whatever we're building in the same way that she would interact with Citibank or else Margo or anything else has to do with money then like right direction so so that's a big number that number is the the total losses in cryptocurrency 2011 - to 2018 across the industry it's it's compiled by a nonprofit called crypto where there's actually really cool little nonprofit they're focused on sort of user security and advocacy in cryptocurrency right and actually I'm my opinion is they're under reporting that number that number doesn't really include the sort of retail scams social media stuff tech support scams so that number is probably actually significantly larger right then then we can track in terms of exchange compromises and major scams I'll give you I'll give you a slightly scarier number right so of that almost 3 billion has been lost a 2011 2018 1.7 was lost year to date 2018 right that is a terror that is the wrong slope for that curve right we want that curve that slope the other direction we want that going down we want less losses over time and so that begs the question and this is really what sort of starts our approach to to a secure program coinbase why is this hard what is so hard about protecting crypto another another way we break down this industry is in terms of causes of loss so this is a graph from the blockchain graveyard if you guys are not familiar with it it's an outstanding resource maintained by a guy named Graham McGee and Magoo who he basically crawls whatever public information there is out there about a breach at a given organization tries to find our root cause or tries to sort the wheat from the chaff and puts it on his on his github repo right and then over time he charged that his think he's tracking he's tracking breaches from 2011 to 2018 really great summary of each of each one if you have if you're interested you have to have I highly recommend if you're in the industry you go read it because it is it is a list of lessons learned in our industry or put another way a list of things to avoid in our industry that's that's really really important blockchain grid just Google watching graveyard it's the first result kind of a unique to so so this is total he's tracking in there 59 breaches since 2011 that's that's an average by the way to save you guys laugh of eight breaches a year in this space incredibly high number higher than basically any other industry I can really think of besides maybe the payment card industry but that's such a huge industry that the numbers don't really quite quite compare so we ask the question again what wouldn't we ask the question again why is this so hard this starts to give us some answers give us give us an insight into how are we losing right and the interesting thing to me is by and large we are not losing because of esoteric cryptocurrency realm abilities as industry we're losing because of bread and butter server bones at bones you know not oh not enough customer off we're losing because of scams we're losing because of security problems that we as an industry been working on for decades at this point right what does it say down here protocol cold I think there what is that but we're breaches of the 49 that he can actually point at a a protocol level bone or or something like really really deep in the cryptocurrency world so the
we go back they get into to sort of why is this so hard if if server breaches are are the cause like why haven't why we solved this problem yet right it won't anyone tell me what movie that's from yes outstanding give that man a cookie which tired yeah there you go knocking about yep so those are those are bear box those are 100 that hundred thousand dollar bear box right and and when I think about for the the security model analog for cryptocurrency but I think of is digital bear box a lot of people say cache I don't like the cache analogy because if you've ever tried to move ten million dollars of cash it's actually not really easy it's a very heavy and bulky and and not easy to move around bear bombs are super easy right that's dock right there's if it was real obviously it's not but but is easily hundreds millions of dollars yes sir yeah doing something that was awesome I expected no one to get it good job I made four more these out there now she had more screenshots fail sorry so so why is it like a bear box so I'll quote you from Wikipedia the bear pond differs from the more common types of investment securities and that is unregistered and the records are kept of the owner the transactions or ownership however physically holds the paper owens the bond is that sound familiar to anybody that sounds a lot like cryptocurrency to me in terms of the through the the threat model for a theft so so what we are trying to do is protect an asset that's globally valuable a bitcoins a bitcoin no matter where you are on the world that's digitally transferable which is actually fairly unique and it's irrevocable if we set up to design an asset that people would want to steal I'm not quite sure we could have made a better one right I I don't know what we'd add to that to make it more attractive to two attackers so that's why it's hard alright we're trying to do and protect a new asset a new thing I'm gonna leave it son at the end if you don't mind well slide so like I said really what is
a new class of asset alright that has funded different risks from previous assets its Manila we were doing that in the context of an online service right it's not like this thing's in a vault somewhere necessarily most of us in the industry who are building these systems are building them in a way that you know users are interacting with them on a website on exchange on a wallet or whatever right and because we're protecting a brand-new my asset class we as an interviewer sort of learning as we go some extent here right what works what doesn't work and how it works especially around sort of the the areas of protecting these assets that are don't quite fit the mold of anything else in the world so so we're sort of inheriting the threat model as an organization in this space of sort of part maybe social media company part bank and part something else that's like not yet defined right so we're trying to fit solutions and and controls into a framework that is Newton and and for we're naturally having having teething pains doing that as an industry so no wonder this is such a hard place to exist so great it's hard shucks I'm sure all of you are shocked to find out that cryptocurrency Tiffany cryptocurrency is hard okay so what does it look like for us to defend this stuff right so so like I said in the beginning I'm not I'm not gonna go over like coin basis entire security program we'd be here for a week but I think there there's some stuff to talk about that that is interesting and unique and how we approach this problem and the first thing and actually all talk a few these entering things we actually open sourced a fair bit of this stuff already and the stuff it's not open source most of it is moving that direction of the next six or nine months I'll highlight sort of what bits are open source what bits are coming and what it's you can learn more about in other talks the other thing one of our sort of foundational ideas here is that is that trust should be created through transparency right not through not through blind faith alright so if I'm asking you hey you've trust me with your money I should be backing that up with a and here's why all right here's we're gonna do to protect it and make it safe and keep it safe so we spend a lot of time talking at conferences and events about or the tools the techniques about what and why and how and where we do it and and our attempts to continuty of it and do it do it even more so the first and most important thing to think about when you think about what Mesa security program is is the people right so today coinbase is call it 500 people security coinbase just 30 people that's six percent of the company is focused on security which is insane ratio for most organizations most most industries right and I think the especially when we're talking about an asset like cryptocurrency right where there is a ton of innovation and we built a lot of our own tools we are really forward-looking you have to start with the people because there are the ones that are going to innovate they're going to find the new ways that they think about securing this stuff that are that are going to actually be the ones solving the problems you I can't look at a vendor for this there there are no vendors that look at protecting wrong there probably so but there are no vendors that say you know I'm gonna protect your cryptocurrency and you put it here and we're gonna make it make it safe and make it easy so it goes back to the people and I'll say I'll just say once again and toss it out there we're hiring just saying some people over there you can talk to if that's interesting so this is a pretty picture but we're not actually gonna talk about but it's very pretty this it's it's a high-level architecture point base very very high level so high level that's not actually useful but it's pretty sure I have it so so if we go back to that that watching
graveyard slide for a second number one leading cause of these tract breaches is is or these tracks sort of losses a server breach all right so when we think about our security program we walk through why is this hard one of the
answers is you know what attackers are walking through the front door let's talk a little bit about how we how we close and lock the front door in this kind of environment we got claymation encrypted currency in general I think we're actually super lucky because we don't have a legacy to deal with for the most part all right we get to build this stuff from from the ground up and we get to build it and I see this as lucky others will disagree we get to build it under constant pressure from attackers right my philosophy here this is one of the reasons that when I when I look for places to go I look for places that great attackers no one teaches you like an attacker right you you you never innovate as well as you do when you have a clear and present threat or danger to innovate against all right it motivates you it focuses you it makes you it helps you do your best alright so we get to build this ground up new technology under pressure under focused attack right what better place could we go to build something well people look at this and say like wow super stressful it is right but it's also amazing so so so what have we built here so play base is a fully containerized every single service and coin base is deployed in a container virtualize were native us continually and immutably deployed service let me break that down for a little bit first of all this is all based on a custom Orchestrator we built internally called code flow which are open sourcing piece by piece we we open sourced our actual deployer itself called Odin three months ago it's a thing that takes a description of a deployment it's a JSON file and actually makes it happen in AWS and it would be bored it's a it's just like the rest of point of a service-oriented architecture so we're open sourcing piece by piece of code flow you know as we quite honestly as the development teams are happy with with it and want to want to actually get it out in the world so code flow handles code from from PR to deploy and prod right it handles the entire path and manages everything from consensus requirements on code submission which we'll talk about more in depth to security scanning to see ICD to build to consensus on or to to your secrets management to deployment it's all in sort of one one long CI CD path but that what that means is that we make we can make a lot of this stuff transparent to our developers build or really overall very very safe and secure CIC pipeline which all gonna hear a little bit deeper a little bit later so so let's dive into that the rest of that so containerized right like I said every single service syncline bases in a container some of you were probably shuttering when I said containers because you have some sort of container here containers can be the best thing in the world of the brush thing in the world when it comes to software management and deployment it can be the worst thing in the world when you back into it right as you discover six months after the fact that your dev team is using containers it can be the best thing in the world if you walk into it eyes wide open and say you know what we're gonna be able to use containers but you know what we're gonna make it easy for you guys to use containers we're gonna manage a bunch before you sew a coin base the infrastructure team manages a baked container that all containers and corn-based must descend from we're not pulling stuff off of docker hub it just won't work in our environment developers development teams then base their services from that set of base containers so that we control the underlying layers we can do the patching we control what tools are installed whole nine the developer divining service-specific docker files as well as those of you have some container exposure now you can do a lot of damage in a docker file as if you if you you know the most the most obvious and consistent example is you know a beautiful base layer that then has like a curl open SSL point you know 0.9 directly on the file systems library right that's not great and it's very very difficult to detect that in a lot of cases so what do we actually run our docker foster a lender and say hey development team you can't actually do that you can't use run and double you get in the same line I'm not gonna let you do that alright we're not gonna let you shoot yourself in the foot but instead here to go to go through this paper of this other way of doing this will give you the packages you need that are up-to-date that we can track an update and make better write and and that not only takes load off more developers but it means that we control our environment in a way that's really hard to do outside of that kind of setup we control every single package every deployment every line of code every version we know where it is when it rolled out where it rolled out how it rolled out right that's also from my perspective so you might ask okay the containerized the virtual I just like what what right why why do I do both what's going on so there's this sounds like a train is arriving outside here so there's two reasons for that one is we when we think through this or the threat model on cryptocurrencies one of the things you quickly come up with with this again this digitally transferable non revocable will be Val currency is that this is one of the relatively few in my opinion industries we're dropping and burning on Oh today actually might make sense right most the time probably economically doesn't work out in terms of like risk of risk of loss sources risk of gain here it might so so one of the foundational things we looked at is we should always this is this is sort of a common fighter gate right we should always layer our security so how we do this is we when we this is also one of reasons we will code flow because nothing else can do this when we deploy right we've your containers on on verts that are mutually trusting right where if you popped one you're probably getting those anyway because there are credentials sitting in you know sitting on that one or otherwise it's highly likely you're gonna be able to pivot we then ensure that containers that don't have an a mutual trust relationship never exist on the same virtual machine all right so that means in order to hop from a front-end web service to a back-end payment service right you're not gonna do that through a side Channel and the verb etre through popping the you know today probe esque in the linux kernel you're actually gonna have to do the hard work of moving through my environment and pivoting where I can see you right not trying to do it in memory on a Linux system the other the other way I think about and we think about defense a lot is you know there's a common defenders have to always be right attackers have to you know be right once which is which is true as far as it goes the other way of thinking that about this this sort of setup is that attackers have to play on my playground they have to come to me and exist in my environment so then my mission is in is to make my environment is in hospital as possible anyone who's gonna come and try to take my crypto right and if I if I do my job well I can I can make that actually quite frustrating so Boop that was the wrong thing so then at the end let's talk about that that continuously deployed immutable bit right so we deploy on average there's we've published some blog post about this so we can go actually look at the data something like 20 times a day are we deploy a lot every time we deploy we are rebuilding that service from the ground up with no overlap new verts new containers new security group anyway SG whole nine yards there's not even any network connectivity between the two services if we're deploying 20 times a day with that kind of a rebuild sort of structure built in that means on average I think our average lifetime is something like 1.5 hours for a service running in prod that's a fairly frustrating environment for an attacker to live in and really what it does this is why I particularly like it is it makes the attacker we exploit every single time right exploits are inherently unstable especially remote exploits you're trying to land on some random maybe this virtual machine in the cloud you're not mean you're not sure what actual operating system is underneath there you're gonna crash stuff if you're reading at that frequency even when you crash stuff even if you're the rest of your exploit was completely stealth I'm gonna know because it's gonna it's gonna crash right can we track that so that I can come back and figure out what crashed why what's going on right and respond this goes back to you as an attacker has to live in my playground that means I get to set the rules let's talk about app sector for a second we will Bunch Minh I think the focus is in our program again walking back to this phone graph right so right after
server breach ignoring unknown because unknown is unknown we have application vulnerability okay so we spend a lot of
effort on making sure that the systems and services we deploy are our safe are defensible are tested are documented or a threat modeled and that fundamentally we understand what we are running not just so that we can defend it but so that we can act to prevent anything from happening in the organization so so one of the things I'll walk down this this overall thing the the first roll star was Salas so this is all tracked open source I would I would guess probably ended the year so Salazar is our static analysis framework right so so what we did early on was was looking around we had the realization of course that hey you know what if setting inside the server breach thing right attackers have to come into the app by and large so how do we make sure that as we're shipping and we're shooting very fast right 20 times a day as we're shipping these updates how to make sure it's safe so the the one of the answers to that is through automated analysis of code before it hits prod in a way that gives engineers immediate feedback as to like why if assuming we flagged something why we flagged it what's wrong with it what they can do to fix it and hopefully not do it again and in a way that aggregates those stats over our entire base of developers so we can look for hotspots we can look for issues they can look for teams that need a little bit more engagement right so that's Alice what's Salas now one method we could have done here is let's build a sack a static analyzer right but that doesn't make sense to me because there a lot of great stack analysis penalize ours out there already what what's missing is the ability to point them of what they're what they're best at and normalize their results in a way that we can then use that one stream to say hey they here the problem exists here this this is this is what the problem is that's what Salas is Salas is essentially a framework for interacting with stack analysis tools that can pick in or out based on the language it detects for a given project pick the analyzers that are relevant for that language in that project take the results back in whatever format we get from the random analysis framework we chose to use fat language with that problem put it into a common format across the board and then use that to interact with the original pool request and say hey line 15 this tripped this rule from this this analyzer hey developer please please resolve this before you're allowed to deploy and then Salas will say and by the way you can't merge this change until this is resolved no humans no humans in the looping interaction of our development teams get security scanning they get instant feedback they get feedback in line in a way they would have gotten it for in a code review and we protect coinbase itself from whatever that that change may have been it's a really really cool little tool I'm gonna be really happy when we get it out out in the world this the second thing we focused on again was this was this idea of sort of a the the CI CD pipeline right and if we're integrating security into an organization is moving this fastest coinbase we have to be as close to zero touch as we can possibly be right so so this this mirrors a little bit of what I talked about for the infrastructure side of things right about that we want to integrate with development with development practices seamlessly right we want to be there as part of the CI pipeline we want developers to find it easy to write security unit tests and part of part of following up on any incident is how do we make sure this doesn't happen again so we want to be there help them say you know what it's been a secure unit test in this in this CI pipeline we won't let that deployment pipeline to be easy and straightforward and have developers not have to worry about managing their systems because we can manage that for them right consensus requirements are code Marge this is actually I think a pretty nifty thing that we do so to push any code change out in coinbase requires sign-off from multiple engineers right and so you you want to push a change to two to coinbase you're gonna have to do you with your selfish submitter are gonna have to go get call it three other people to say you know what this is a good idea most people can't have can't have added code to to the actual PR question asked me to three is totally independent people as you do that that plus one it's just a plus one in in on the pull request it's to factor oft right so you get that nice push to your phone that says hey you just said we should deploy from service X commit hash why was was that cheated you mean to do that and soon the answer is yes then great everything emerges it gets all gets deployed assuming the answer is no that kicks off a sort of a minus one process right where we can actually say you know what we argue actually need more eyes on this this three is like this is this is super terrible you should have required four or five or six reviewers depending on how many - is the the porcfest is kidding all right this lets us ensure that no one individual is another sort of core concept we shoot for no one individual no single person can make can do a thing that impacts coinbase or are the the PII the Fiat or their capital currency that we have we have stored it should have always require a conspiracy because conspiracies are fragile and they're scary and they're really high risk for the conspirator and we want to again the attacker in this case has to play on our protector playground I'm gonna make that playground rule as hard as possible for that attacker the concierge OPSEC program so so this is one other this is one of the reasons the apps that came is probably my biggest team according this is because we want apps tech not not to be the guys that sort of parachute at the end and say no everything's terrible let go home this is this is crap you can't deploy it that's not productive for anybody instead we want our OPSEC team to be in the stand-ups of these development teams to be in to be in wedded at the hip to have all the same incentives to get where they're trying to do so that as their as that team is writing code is making changes that app SEC team member knows exactly what's going on across the board and can render for that independent third party oh hey that's actually pretty risky you know and like let's let's let's engage let's how about and that's that's making this change again the this this is the the sort of the the app level vol and it for an exchange especially is is one of the most worrying things that we deal with we want to make sure that we're providing that expertise and we're providing it in a way that developers wants to use it right that want to get engaged the last thing I'll highlight here that I think is quite cool is that that we're getting much much more into recently is blockchain monitoring right so if you imagine a world where ever there's you have to imagine it it's today what are there 18 hundred or so assets somewhere in that in that in that vein in that vicinity 1800 crypto assets and in the world today as as we think about you know looking at at assets that you guys so we just added et Cie we've talked very publicly about about the desire to add more one of the things we want to be sure of is as we are those assets that we're proactively looking at those block chains to say is this assets still safe right do we know it's one of this asset is there 51% attack happening is there you know is there a contract indication of a function a token that should never be in that should never be invoked so
that we can take immediate protective action right all our systems so we're protecting the extent we can never borne abilities are our assets or our customers assets more or more appropriately this is also an area that we're pushing pretty hard in and I hope will opens we'll open source some tools in as we as we get that stuff built built and roll it out because like I said at the end of the day what Trust should be based on transparency you guys shouldn't have to wonder how in protecting your crypto you should know let's talk about the texture spots okay because at the end the day you can't wait you need to have the ability to detect when things go wrong and you need to have ability to respond safely right so so that goes to the heart of this first one it's my my favorite name of all of our projects in Dexter it was our he's our friendly forensic assistant few Dexter fans in the audience I can see outstanding the rest of you can google it later and you'll laugh when you do it so so what'swhat's Dexter why do we build it should go back to what I said earlier right no one person should have the ability to impacts or store or steal you know coinbase crypto VIII fiat etc but when you think about Incident Response that's one of those areas where the response is frequently Oh we'll just throw all that out the window and we'll just go ahead and respond as an individual right that to us is is extremely dangerous and worrying we don't want that right but at the same time we want to be able to respond to incidents quickly and a very very agile ease how do we how do we square that circle what what Dexter does is is two core things number one it provides a consensus based approach to executing forensic commands right so something happens some instance is doing a weird thing in in production and maybe it's an instance it's actually you know dealing with dealing with with crypto right I don't want an instant responder hopping on there but the instant responder can spend up dexterous or an investigation and say this is weird giving a process listing give me you know basic live response processes LSO F maybe some stuff from proc anything what the actual problem was and that investigation spins up and then it sits there and waits for a plus one from another in response engineer depending on the commands you're executing the ax Dexter the number of required reviewers changes alright so maybe process listing l so I'm not a big deal that's just a plus one maybe he then says you know what I actually need a memory dump because I don't know what's going on here I need to really get deep into this that's gonna require a lot more consensus right maybe requires a plus two plus that he maybe requires sign-off from me or maybe requires sign-off from legal right but it lets us very flexibly define who and how can execute sensitive commands even in a fast moving hide the Critical Incident Response situation right we never want to be back to that place where one person can do a thing it's also built to be sort of there's a bunch of other cool thing about it around in our in our 80% of a command it's it's highly segmented right so so hosts normally can't talk to each other's there's no place on our network where you can go talk to everybody to do internet response right so it's s3 based so we're polled we're pushing into a cue the hosts are monitoring their pulling down and staying up is this for me is this for me yes no maybe it's all backed by DBG signatures right so we're actually basing the stuff in crypto not on like a software counter that says oh yeah you post one me enough I'm good to go this is this is actually so the guy he wrote this is open sourcing it at Derby Con this year just got his talk accepted two weeks three weeks ago if you're a Derby con you should really go to the talk cuz it'll be awesome so the second thing I'll talk about here is is again another another derivation that of that container container base stuff so I I do another talk you can go find from Chaka Khan I think was the last place I did it it's incident response and authorized and containerize environments where I talk through your sort of what's unique and special about about containerization when it comes to incident response to detection it's it's it's a really cool interesting environment but the thing we'll highlight here is that because we've docker is everything everything's in a dark container everything is running isolated that means we can very easily pull specific behavioral information about a given service right so the reason that this stuff doesn't doesn't work well across the board normally is because they're just so many signals to deal with right there's system level signals there's if you try to do this on laptops god help you because that's even that's even worse but even on the server environment right it's really really hard in our environment we can constrain it down to our docker container and say how is this docker container behaving how is this single process acting and is it acting different than its peers right and then based on that we can we can do things like you know using core basic tools all 2de BPF look at that and say how is this acting at a system level different than its peers is this is this behaving in a way that's indicative of an attacker being on this environment and if so we go back to dexter and we can actually respond very very flexibly to that to that area figure out what's going on and sort of figure out what we need to do next the last thing i'll talk about that we that we do that that's that's cool and special is we log everything when I say everything I mean everything we do this for a couple of reasons never want to go back to that deployment cycle we're to put forward a point point times a day average lifetime is like one and a half hours what happens if there was an incident and we discover eight hours later right container is gone what I'm gonna do with it all right so the answer that is we log and and in much most importantly enrich everything immediately right so when that container is going I can still walk back and pull those pull all the law because they were issued by that container tagged with that containers name and version number tagged with you know the the processes the result process names whole nine yards most of the stuff I would get from alive from a live response I can regenerate from log we maintain of these of these instances and I can get that I can get it quickly I can get it searchable right the most important thing I think the most important and different thing we do here is metadata metadata just all the metadata anything it might possibly be useful in the future we'll tack on a walk up time we tackle blocks so that as we roll into this into any future investigation there's never any to the extent we can any ambiguity there's never any question that we have the data we need to make a determination as to what happened so we've got about I don't know seven or some minutes left but for all open up for questions before we do that I want to talk a little bit about for what we can do better as a community I think there's there's three there's three specific points on it I want to drive home right one is today I just that real there's not enough very intelligent staring in the community there just isn't we eat we're we're fragmented ideologically or fragmented commercially and we do not talk to each other enough even though we share the same opponents the same attacker coming after me is coming after Jim and I is coming after anybody else here who's running who's running an exchange or currency we shared opponents okay why wouldn't we share threat intelligence about those opponents to make us all stronger so two with one concrete thing here I'll just all flag really quick is we recently in working with some other
exchanges and some other traditional financial folks started an FSS act working group you don't know what FSIA is rights financial services information information sharing analysis center it's been sort of a cornerstone of what traditional finance how traditional plants has done this for decades right this is sort of a nonpartisan not inviting one company a way to interact and share with peer organizations so we started this working group and we're reaching out now to to you learn everybody who who's involved in the crypto space to say come join how Plus establish this sort of basic level of security trust and cooperation so that we can all get better number two and this is actually also related to FSI shack we're we're not really good at you know standards at the community there are a few out there right the CCSS is one that's been out for a while and gotten some traction but when you go back and you look at sort of sort of how traditional finance evolved over time I'll give an analogy that I like see if you guys like it too if you were in the 1800's 1850s right walking around looking for a bank to deposit your money you're walking town Walker to bank this is nice this is a place this is great one of the questions you probably got asked tell me about your vault right how do you how are you gonna secure my money because bank robbery was much more coming there are more risks everything was more fragmented and as an informed consumer you needed to ask that today it let me ask them and should pull the room has anyone ever walked into a bank and said before I deposit my money I need to see your vault the only anyone ever asked that no you know why because that industry grew up they've built standards it built and through your standards that built trust with people right it's the more complicated story there's some assurance involved there's regulation right I'm simplifying it but through the use of standards we can build trust trust with customers trust with regulators right so that the question shifts from from where it is today for crypto I'm gonna deposit my crypto tell me about your security to where it needs to be tell me about your services tell me about how you deliver a better service than whoever else tell me about why I should put my money here because I'm gonna get these other benefits so the ask is he didn't under the auspices of the FSS like working group or or or via an organization like the CCSS or whatever we as an industry need to invest in standards we need to invest in not just creating standards but coming together around standard to driving ourselves as a community holding ourselves to that standard so that we can be worthy of that trust last and this is obviously self-interested point really not enough folks in the security community are looking at crypto and saying like you know what that's a great place to do security right you know we see some of it it's starting to change certainly over the past year I've seen it I've seen a change but but I don't think we as a community are doing enough to talk to the rest of the security community not just about how cool crypto is because it is really cool and that's but about how what an interesting problem is that this is right oh interesting is it to secure a thing that's never been secured before to write the book on protecting an asset in this adversarial environment right people should be knocking down the doors of cryptocurrency companies to say wow that's a--that's a great challenge like why wouldn't I want to engage there and I think they're not because we as a community again we're not doing the outreach we're not talking to the rest of the community about the interesting security implications of this space we're talking about how cool crypto is this is all so fishy to say if you're known for corn base come talk to me but you know and with that what are your questions I told you pick who who gets first are making that comparison because anyone who has access to a private key can immediately gain access to the funds so the question is how does that differ from possession of just a password to a bank account because there are other controls there right there the you so a bank would have the ability to do fraud detection on transactions the bank would have the ability to do locate geo determination you're in you're in the US and there's a login from Ukraine like maybe we should do something about that right there there there are other steps you can take in the middle there as opposed to possession or private key possession of a bearer bond that's it game over done it's like a debit card for example anyone who extracts the money can just run away with it sure but depending on how they do it where they do it right there it's if you have a debit card that debit cards connect to a bank account or there are controls on the interaction between that debit card that bank account you can't the debit card doesn't necessarily mean I get all the money in the bank account right the because there is a control environment between the two there are daily limits there's the ability of the bank to say this is all transaction I'm going to free as this debit card right there the the core of the comparison I think is once you have a bare bottom less you have a private key there is nothing anyone else can do to limit or protect from the loss of that that value thank you yeah hi I have a couple of questions about custody to have a like a compliant custody solution number one is it necessary to provision identity over the account so that you know exactly who they are secondly for custody is a centralization essential to secure custody and thirdly when it comes to the regulatory regimes that make the I guess create the most work for you I mean is it gonna be the OCC or is it gonna be the NY DFS like where does most of your as far as complying with regulations who's who's the toughest regulator okay so the first question is is really a question about sort of BSA right and and a more practices and in general yes we have we have to for us consumers you have to walk through BSA requirements we have to do kyc on customers before before we're dealing with with their assets for the second question on or is it centralization on security um I I don't necessarily in the abstract right because it's security is is can be defined relatively right so as certainly an individual can can invest in a level of security that's that's good for their needs and their throat mobile without ever having to put assets anywhere right and a lot of folks do you especially in the crypto land I think the the other side of that coin is people choose to use a centralized service because we can invest in centralized security or they don't want to spend the time and effort to to build what to them as acceptable sort of security arrangement for their crypto
yeah I mean did we're a centralized
service you know and then the third one about about regulators you know I really guess I see the space as the regulation is evolving and and working with regulators especially coming from a company like like coinbase who has always been very sort of regulation will forward its what what we always want to do is is work with them and educate them around how this space should be regulated in our opinions I I think you know calling calling out any one regulator as hard besides being unwise just generally also doesn't it doesn't make sense to me because all the regulators I've worked with have come to the table with with an open mind about about crypto and wanting not just to make us go through a check the Box mr. sides but actually wanting to go back and forth about about a new asset and and how the old framework should apply to this new kind of asset yeah well I just I've just read the NY DFS particularly the cybersecurity rule and the new AML transaction monitoring and filtering rule that that's becoming I think you know to the extent that that it impacts everyone else obviously it attacks us because we're operating like a potential institution are you talked about the mist yeah after the work yeah you talked about the mist model security a detection like you mentioned containerized what segments the network mom so the difference is in is in the the layer of granularity right so so container has workloads your your the security now is is at the sort of the process the workload layer as opposed to network segmentation right where you're segmenting at the at the server or at the host girl group of hosts layer like the the layer of level of visibility is different as well as you know your the actions you can take are different right so it's it's to me I I personally prefer the segmentation on both and but if I had to pick one I would pick a host because you get a much richer data set out of a host based detection then you get out of a network based detection now pretty much all just yes yes so zipper trade data is actually one of the most important things that we look at we think you know the the last thing we want to do is expose user trading activities both in terms of individuals they don't we take individual privacy extremely seriously in that sense as well as like folks that are active traders with specific strategies are executing the last thing we want to do is be in a position where we're leaking data on those strategies in a way that could be taken advantage of by by others so we did thinking about sorry thinking about a better environment a little bit control of any sensitive a like that is really predicated on making sure you know who is accessing it for what reason in a win and why and having the ability to flag deviations from from any sort of normal patterns there which is at a high level the approach we take great with very restricted access lots of monitoring on interactions with that those kinds of data sources and sort of active oversight by the security team on that kind of activity high level sorry
I'll just our call analysis if we have no longer mics you and then the drum in front of you yeah so so we do use dedicated instances in areas where we think the workloads security or the workload sensitivity merits it the the second piece there and I've had discussions with people about this in both directions but because we we move so rapidly in in the environment is it sort of a side effect of our deployment strategy it's actually fairly hard to end up on a host with one of our systems reliably right so like the setup for that attack is is pretty significant German the white shirt and then and then black someone talked about custody and regulation earlier and one thing I wanted to ask you is that many centralized exchanges have are in the process or have launched to 0x like exchanges like we see on chain custody what's your what's the house you yeah so we acquired a company called pair Dex what was it three years months ago that's operating as operating at a zero X based relay right so yeah and from the regulatory standpoint do you have any remark on that or not really I'll say that that it's coming back outside of the u.s. first yeah John here yeah that's an interesting question he actually offered a multi-sig wallet for a long time had an extremely low uptake rate from from customers right the so low that that it was wouldn't when we did the the evaluation on should be should we keep this feature or not the the code we could simplify by taking it out was was you know a better trade off than then keeping a feature that almost nobody used right so I think if the customer demand is there sure but we just don't see it do you think what what needs to be done for coinbase to be able to accept an arrow and other similar strong privacy corns that are out there fair enough I actually am surprised that question took that long look I think we've seen some really promising moves from regulators and put the privacy points base Jim I in particular has made great progress with with easy cash and and getting regulators comfortable with the privacy coins I think for us right there's the the primary thing that we want is to make sure that customers on our platforms are getting access to the assets that that they want and that and that they want to use so I think that we we look at this primarily in terms of a in terms of our digital asset framework that we've published and said hey here's how we look at digital assets but also in terms of what's actually going to be most useful for our customers and what if whether they wants the most and then we figure out take a look at like how can we do that yes yet just a follow-up comment on that the reason that the die-hard bonds were valuable is that they're fungible also I had a question on your on your previous slide which makes it look like the be like the hot wallets you guys named that cluster Knox uh-huh is that actually true it seems a bit obvious yeah yeah okay is that is that a comment on our poor naming schema or we like you see like simple names short names Dexter Knox in the corner there I think so one of the things you talked about was the various things that you've built internally and are now in the process of open sourcing so sort of in the build versus by open source world it seems like the period immediately after open sourcing an internal product is sort of the most dangerous period like you don't have you yet to get people fixing things externally but you know it the codes out there now uh-huh so what are you doing to defend yourselves from sort of the the potential zero-days in your own software kind of a scenario yeah your own internal vulnerability scanning for those types of systems yeah that's one of those very broad questions we can be here we can answer but we don't treat internal software any different than external software in terms of our overall OPSEC program right so so code flow is going the same there which which is the example I use it's going through the same scanning tools the same apps that process the same the same testing the same everything as a public-facing service that's that's part of our our overall point based services probably the the the second part your question is interesting is like the risk trade-off between exposing source code and and not exposing source code I think it's a really interesting and sort of nuanced nuanced trade-off to make between this sort of principle that like trust trust should be transparent and and risks we have around releasing code that might have issues in it we in particular from the open-source components like like Odin which we did several months ago we we spend a lot of time and effort looking at that before the release we tend to release small pieces of code as opposed to like here's a hundred thousand line behemoth we want to release here's a five thousand line tool right because we're much we can we can we can't put that much more effectively in terms of app sack yeah so it can connect to s3 and then the endpoints Pole s3 for information right so it's not it it can't connect to anything it could just talk to my sweetie bucket I think we you might have time for one more question if there's a bit more time if you have time yourself and we have nothing scheduled for 11 o'clock so if you wanna I'm like gonna stay up here for an hour if you want to do a couple more that's okay there's supposed to be another if you don't let's find two I'll do a couple more and then and then and then get out of here you mentioned intelligence sharing and FS Isaac is a working group I'm curious about like how valuable that data is for you as like probably your tech stack is completely different Wells Fargo sure and and that's why I think this is so there's there's two pieces right one is the financial actor sharing and one is like sharing with encrypt oh so two inches that question one is yes my tech stack is probably totally different attacker behavior is probably not that different right how they the kinds of things that they target how they move internally how they act things like that that's what I don't really care about IPS and hashes right that's that's good but it's not what what I really want what I really want our attacker behaviors because that then feeds into my roadmap right attackers want to do this how can I make that hard
frustrating annoying prone to failure
there's nothing else I'm gonna call it gone
[Applause]
Feedback