Logo TIB AV-Portal Logo TIB AV-Portal

Systems Thinking for Participation and Security

Video in TIB AV-Portal: Systems Thinking for Participation and Security

Formal Metadata

Systems Thinking for Participation and Security
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Participation and security are both emergent properties of whole systems. Both properties are closely related and critical for understanding or building tools for humans.
laptop code NET time sources sets bits part likelihood Computing computational management processes hypermedia Computer animation system Security firmware
Context goods processes Lecture/Conference time system similar sets cotangent platforms form God
category building Meeting/Interview hypermedia interactive system website structure platforms
fem building Lecture/Conference Development system system life box game
man building Context ones maximal sets category means Lecture/Conference system testing complex systems sort mathematicians web
man words Computer animation code Development single system boundaries DoS Arm van
code information systems chain system boundaries structure perspective equivalence
category Context Meeting/Interview code characteristics crack system Security Bugs
man curve category Slides Context means Meeting/Interview Gamma Security conditions Bugs
category time system Bugs product
odd maximal system part Security
domain Computer animation code different views system circle lines Security Gravitationsgesetz
category rates Meeting/Interview time administrations system part Security computational
Actions states moment interactive part open subsets computational Facebook's rates Meeting/Interview Lecture/Conference negative disk system structure sort Security spacetime
man loop closed interactive knot system effects utilizes part measures Twitter
man rates Meeting/Interview code system bits Security van
complex category programme pixel crashes Meeting/Interview time interactive system Security theoretical
hard complex default building mapping load decision time graded routine limitations bandwidth Lecture/Conference system Combinatorial
man processes load Meeting/Interview time routine interactive
man noise Actions oriented decision forces model neighborhoods storage sets plan similar equivalence loop Meeting/Interview system Right Security position spacetime
man environment Meeting/Interview interfaces time traction traction van tasks
dynamic Context Actions Meeting/Interview traction system part Security modes
processes Lecture/Conference hierarchy time decision interactive normal system structure Security powerful
services Lecture/Conference Average capture system sets Security number tasks
web pages man Actions dynamic Intranets information Meeting/Interview different interactive Right instance
relation different time web pages Content sets similar ideal cycle
information Lecture/Conference surface interactive Right Security frame
flow Context loops Lecture/Conference phase system sets Security Jump form tasks
domain Context Software Meeting/Interview Lecture/Conference time level system physical part Security
domain complex Context Actions building information time sets mathematics invariants Meeting/Interview Lecture/Conference different touch Authorization level system Right Security conditions tasks
building code instance part mathematics Meeting/Interview Lecture/Conference terms system life pattern structure Security tasks
time catalog student part system call rules category Meeting/Interview system pattern Right structure Security
Context time fit experts Content bits instance Benchmarks van goods processes Lecture/Conference Meeting/Interview website system game fitness
man dynamic Context real time law sets solid number goods Lecture/Conference Meeting/Interview square system sort structure table tasks
Lecture/Conference internet square system Right sort number
man flow Context Java interactive sets iked processes Lecture/Conference Meeting/Interview system Right box Security Chi-Quadrat-Verteilung
man Meeting/Interview interactive system van product
interactive maximal sets van invariants Lecture/Conference Meeting/Interview testing sort game Gamma Security Results
area time real Development sets transfer distances category Lecture/Conference different system testing Security
time Development interactive sets bits processes rates operations testing structure Free Security write
Lecture/Conference Meeting/Interview time system essence
Actions response plan sets Types processes Computer animation Software Lecture/Conference Meeting/Interview system internet of things position
data management Computer animation Lecture/Conference Meeting/Interview smart system
man inference goods Actions Meeting/Interview Hardware machine maximal life system plan part
man Meeting/Interview Lecture/Conference model maximal system
dynamic time part category message-based Meeting/Interview different Universal cores system Right Arithmetic logic unit Security
but the and sources the
the so and I spent a lot of time hanging out with both the security community and the design community and they don't always get along very well they they both seem to I think that they know how the world works on them and in part this is a little bit of a plea for these communities to get to know each other better and but I think it is actually lot a really interesting stuff that can kind of come between them so let's dive into it a lot of people have this really unfortunate belief that security is about computers and security if she has very little to do with computers as most of the time what we actually care about when we when we're caring about something about security isn't anything to do with what the computer is actually doing something that's happening in the world we don't really care what code is running on our machines and if you did you'd all be rejecting the closed firmware where all the DRM code that's running on your devices on because you have no idea what it's doing all this code you don't actually trust you be really worried about that but you're not on a kid like the 2 of you in the back to them so security is a set of activities that reduce the likelihood of a set of adversaries successfully frustrating the goals of some set of users this is the goals that they have in the world not what code is running on a laptop but rather you know can they actually get the job done what we're actually talking about when we talk about security is efficacy we're not talking about is the system secure we're talking about can the person get the thing done that they thought that they were trying to get done because that's what everyone actually cares about now there's a really
interesting and kind of similar that's very similar set of misconceptions about participation and what it means to participate in the process of using a system 1 of the problems that we have right now is that in many contexts people think that spending a lot of time using a system means that there's a lot of participation going on that it's a good system on how many people here spend a lot of time filling out your tax forms is that a good participatory experience yeah I the 1 of the other things that people tend to assume sometimes is that when you engage with a lot of people and you interact with a lot of people that means that you're having a positive experience for the platform that's great when they're being nice when you
wanted to engage with them when you had any desire to talk to any of these people ever and you're not just trying to get away from all of them 0 my god our but there are no non-participatory systems this is 1 of the other things that people who are designing for like 0 well
it's not a social media platform so clearly it's not participatory that just means that all the participation which is happening which is to say all the places where human beings are interacting with the systems in their lives and talking about what those interactions are it is not designing any of those as just happening somewhere off in the world and you have no idea what the participation structure of your system it's
I the 1 of the other things here ads if they're effective are always poisonous to participation because no 1 goes to a site with the possible exception of under no like a site that is showing advertising awards no 1 goes to a site wanting to participate with an ad no 1 ever wants to engage with the brand and so you can tell yourself that that's what your site is doing but you're flying to everyone and and and this has really interesting consequences the so 1 of the things that many people think is that they're building systems that people are going to want to use that's again it never really true people or building systems that let them accomplish things in the world that they're very interested in accomplishing and if those systems let them
accomplish things in ways that they can otherwise accomplish they may really enjoy using those systems but the goal is never to use the system the goal is to get a thing done in the world you even if you're designing games systems the person's goal is to be entertained not to sit behind the
x box on silica video game reviewer and again but people don't really care about what they're doing in your system they care about what they're doing in the world and often mostly developers talk about the people who who use the system as the users but that puts the developer in the center of the picture of the developer isn't actually in the center of any picture other than their own you're not they're not your users your their Toolsmiths sitting off in the corner building something that they don't really care about that much I
if you build tools properly you make both better users and better communities of people and the way that you need to do that ties in very directly with the things that you need to do to build secure systems in certain ways
so no 1 is interested in abstract systems again there's 2 mathematicians in the room were very interested in abstract systems no 1 else is interested in abstract systems were really interested in systems in the context that they exist in the world because that's where all the human meaning starts coming in the on 1 of the really interesting things about big complex systems like most of the ones that were building is that none of the properties in a systems can actually be designed in a meaningful way they all just sort of show up out of whatever set of things that we happen to build some of which we intended to build some of which we built and then
totally forgot about and you get developers going wait using that feature for what but that that was a debug thing that we meant to turn on 0 announcer biggest OK it uh this is mmWave more common than anyone would ever hope I figuring out what is important in a system is very hard and it's especially hard if you're not looking at the things that that system is doing in the world and the it is very useful when you're trying to analyze a system to to dig in and be like OK what is this 1 piece of this system do under dive really deeply into what this single piece of code dies and then you realize that that doesn't tell you anything about what the system as a whole does and you always have to jump back out and figure out 0 well that means this thing here and this thing means this thing here but the whole system means something else but if you know boundaries are useful boundaries are very useful but if you don't understand what you're boundaries are
doing and the thinking in the work that they're doing for you you're going to fail to understand the whole system this is very clear when we look at
security systems I think that we're starting to see the ways that this is clear and participation systems but it's less obvious the most critical skill is basically learning how to reason across structural boundaries now if if you've got a bunch of code over here a bunch of code over here and you're looking at it from the
perspective of code it's pretty easy to kind of chain across these things and like any programmer or know the equivalent for any designer ends up learning how to do those kinds of this kind of leads across systems boundaries but it's very difficult
it turns out to go from code to design or from design to legal and that's what the bugs start getting interesting that's where the that's for the cracks get interesting so a lot of people and these are
mostly people in the security community are like 0 security is so special security is so different you know there's a reason why we have to be these cowboys who who you know do whatever on security is just another property of a system like weather's performance reliability I any property that emerges from a whole system has a fairly similar characteristics and you can only evaluate them from a given context
and from a given standpoint or in a given context from a given standpoint you know secure for whom secure with respect to what secure under what conditions sufficiently fast for whom sufficiently fast under what
conditions and this is always just a means to an end no 1 cares about performance we care about can I get my work done no 1 cares about security we care
about can I get my work done on 1 of the other things that we find in this is true again of of any whole-systems properties that they're incredibly expensive to retrofit and the slides they don't have in here this is great curve of how much it costs to fix a bug verses when it gets introduced on and it's
somewhere depending on who's dating you look at between 30 and 100 or a thousand times the
cost to fix a bug immediately when it introduced verses once it's out in production I it's very difficult to retrofit whole-system emergent properties into a system after the fact the now with participation it's also kind of interesting you have this really complex double-blind because you also can't understand very easily what a system means to design and participation until it's out in the world we'll get to that later so 1
of the things that people in security often really want to say is like well OK we've got our system and then there these adversaries you know that the bad guys over there and the glasses in that and that the bodies they're not part of the system but of course the part of the system they're interacting with the system the users of your system they're not uses you want the other users you'd like to have interacting with the system but they are users who were interacting with the system and if you don't understand what you do and don't want them to do then you need to include them in the system abused
historically has fit into systems in very odd ways and it's security in very odd ways because nobody it doesn't really fit it's not a traditional security problem it's just like a bunch of people being dicks
and but that's not a security problem even though it's a security problem for the user that like there's no code getting run and then people kind of think themselves in circles but as soon as you start thinking about well what are the users trying to do our system not what code is being run but what they actually trying to do in the world and how can we figure out you know
how to how to improve that experience it's you get a very different view of things there is no harder line between participation problems and security problems they're not in different domains there really the same thing and and a note on the politics side anything that your system does in the world it is you know potentially a security problem for your
users if you a a B and B and to the work that you do in the world is making it impossible to afford an apartment in New York or
Amsterdam then hey you created a giant security problem for your users 1 that you totally didn't think was part of your system but it turns out it is and security is a performance and this is also true participation you can't just set up a system and claimed that it's secure rate you have to continually work to make it to make it secure but for a lot of other kind of emergent systems properties like performance like reliability we've automated out a lot of those issues right no 1 spends very much time now thinking about 0 will have a wide what 0 I have to do to keep this computer running quickly unless you're a sysadmin professionally and but it used to be that you know you'd spend a bunch
of time with light you know and cleaning up disks and all this kind of stuff that you had to do or you're going to have these problems we've eventually matters automate that stuff I this can be done as easily for security for a bunch of interesting reasons that will also get to later the part of it and the most fundamental thing of it is security is always transient right you're secure in a given moment you're secure as long as you continue performing the structures that keep you secure because security is part of participation specifically security is the negative space around participation
right every system that has humans using it is fundamentally participatory everything that you want all of the actions that you're designing to enable those of the participation actions all the actions that you're not designing to enable those of the Security interactions and you can't really you can't really consider 1 without the other but because we had this problem where we thought the security was about computers for so long we sort of forgot to look at this stuff 1 of the places this comes out is the notion of engagement rate so engagement is how we measure participation obviously right if your Facebook you're trying to optimise for engagement on how many people here wrote a book seem like a state anyone anyone I highly
recommend it it's OK a couple its sigh it's 1 of my favorite books for understanding what's wrong with the
21st century if you don't know why people care about your system and people like your system it will fail on there are surprisingly few companies that make social systems that really understand and why people like the system how many people here use Twitter how many people here think that Twitter
corporate knows why Twitter users like Twitter yeah and it's really
obvious we all get we all know very well from our interactions with the system that they don't understand why we care about this thing they don't understand what the real utility is for people and part of that comes from things like this notion of engagement if you because of monetary reasons have to tell yourself a different story about why you're users like your system other than you know why they actually like it your system will eventually fail regular users are all gonna realize that like 0 you're designing for the money that designing for us and as soon as something which is less for effect comes along they will go in and jump over there on 1 of the really interesting things about emergence and especially about emergence when it relates to this kind of you know close loop a close with systems is that you end up getting whatever you measure
right and just the fact that you're measuring a certain thing will shape the system in certain ways rate you you end
up doing an experiment being like 0 this is good so I'm gonna get more of this but you know you
don't really know the thing that you should have been optimizing for and this is just as true for security as it is for anything else but so let's say that despite all of this you would like to try and build systems despite it being you
know what a terrible and complex problem we need to talk about humans a little bit and the way humans actually work before we can understand either security or participation on you no 1 would ever write code for an API that they don't understand but people seem perfectly happy to build
systems for people that they don't understand and people whose lives they don't understand which is kind of surprising I 1 of the things that you need to understand before you start building a system is what the people using the system actually care about again this is not always obvious on this is often painfully non-obvious in systems that are trying to design for security because 1 of the things about security people is that they think that everyone should care about security and it's also true 1 of the things about design people is that they think everyone should care about design so you get these systems that are optimized for every little pixel placement when really people would just prefer that they didn't crashes much so that they could like an no interact with the bank in a meaningful way and and the same thing is true for security we end up optimising for things that are not actually useful for people In theory I mean whatever you know you build a bunch of stuff but it turns out the you know you there are always tradeoffs and then nobody uses your system and then you get run out of money and you know I you need to know what they're actually trying to do and often this is something that you will figure out after you build your system and started using it and people started using it because you don't you don't really understand what you're building but there are a bunch of things that people are very bad at and this is especially interesting for the kind of complex emergent
properties and people are really bad at tracking how different pieces of a system interacts together this is 1 of the things that you know programmers have to spend a bunch of time learning 0 will affect set this thing over here in this thing over here and this thing over here all of a sudden the
kitchen is on fire but none of those were yeah I so any time you're building systems that involve this kind of big complex open-ended combinatorial things like say trust on chances are that you users are going to get it wrong and you need to understand where the limits of your users are and how they think you know how they actually make decisions before you gonna start making those mistakes brains have limited bandwidth and this
is true of any system that brains degraded very weird ways that don't you know that don't map to the way it kind of hard systems the grade people default to routines they default the habits they build habits very quickly so
they they can stop thinking any time you have something really important that you really want people to think about how they're going to do everything they can to not do that because that's not the thing that they wanna think about I there are times you can use those routines in your favor but you have to be very careful than about designing a process of routine acquisition this becomes something very important you know when does the user 1st see is
saying how do they learn that process what is this what is this on interaction look like if you want them to not just do the broken thing again and again and again so the so
called into the loop on 1 of the things that we care about insecurity is specifically how do people planet in the presence of adversaries because they're not just planning I'm going to the store I mean maybe that's in the presence of adversaries reasonably what your neighborhood is like but and the planning and the someone else planning how to possibly harm them so this is originally a came out of uh the air force in the US but observe-orient-decide-act right any equivalent way of modeling how users think is going to go through a similar set of steps and with a similar set of names it is really matter what they are you know I understand your position the world orient yourself within that you know possibility space decide on a course of action do it watch what happens on if you want to add security features to a system or change the participation space of a
system you are adding noise in here right you are making it more complicated for users to orient themselves within the system because now they have
all these extra shiny things that they have to touch and you are making it harder for them to decide on what the thing that they should be doing is you're slowing them down they're trying to do a bunch of other stuff which has nothing to do with the thing that uh that you're trying to add in they're they're
trying to go on about their daily lives accomplish whatever task they started using the system for
so you need to think very carefully about you know is this complication is this tool worth it the stress is terrible for people thinking coherently so I spent a lot of time working with high risk users in very complicated environments 1 of the things that we find in those places is that all of the users are under a lot of stress a lot of the time and that makes interfaces that might be easy if they're sitting at home at a very simple environment and not worried about like the safety of the children are these kinds of things all of a sudden these interfaces are
much more complicated so anything that you can do to make a system less stressful during kind of everyday use is going to pay off when it actually matters and again when you're talking about either participation or security both of your failure modes are quite stressful so anything that you can do to reduce that stress is going to help people I drew
dynamics this is in part here this is starts getting us into actual participation design but there are a bunch of things just think about a new given context how the groups interact how do people on work together you know what are the
what are the traditions were the norms are you dealing with hierarchical power structures are you dealing with non hierarchical power structures and what you actually designing for 1 of the other
things that are a lot of time especially in security also even even participation design people forget is that the the people that we don't want to interact with the system are also humans and you know if you want to design against trolling if you wanna get design against abuse 1 of the ways you can do that is by understanding like what are they you know what's the return on investment for a troll what are they actually getting out of this process what they trying to do what is the decision structure around when they're picking victims were picking what their interactions are going to be and disrupting that
you know you can design against these things as well as designing for them so let's say
you still like to help and you wanna build things
1 of the places to start is figuring out what you're going to build an the again this turns out to be somewhat complicated given the number of tools that people build which aren't actually wanted by anyone on there are a bunch of fairly traditional tools which are not going to go
into any kind of depth here on looking things like personas and scenarios right let's look at what the average set of users shouldn't be using the system let's get ourselves little sketched introductions of them and you know let's look at the scenarios under which we think people might be interacting with the system on you know task breakdown service touchpoints scenario flows you could have had this traditional toolkit for design which you know that captures most of what you want but there are a few things for both participation security that really doesn't capture so our personas are great as
far as they go but 1 of the things that they do not do is capture interactions they capture individuals they capture some of the tendencies of
individuals but the dynamics of how groups of people interact so for instance at 1 of the examples they use with this stuff is you've got a corporate intranet right you've got a bunch of different people who are all just normal users of the corporate intranet there might be 1 or 2 personas at most between them they all have the same roles in the same and from an
privileges informations on the site of but some of them are going to be the people who start
pages right you're just like 0 that's a great idea I'm going to go right this up and some are going to be people who are never going to start a new page but are going to look at a pace
that somebody else's started in kind of add a few things in and you know some of the people similarity to the people who will go through the whole cycle is clean up a bunch of stuff they don't really ever add content now those are all the same persona those are all the same role but they have very different affordances and very different sets of things that you need to design for
i any time you're trying to understand a participant uh participatory relationship you want to keep the data In the relation and not in the end points this is 1 of the reasons why I think the personas are our slightly missing the mark they encourage us to think about these users individually they don't encourage us to think about the relationships by the fact that they store
this information separately 1 of the things that we care about when we're looking at creating participation frames creating structures for
interaction is making sure that people have the right kinds of alibis the right kinds of things that enable them to do these interactions and those alibis are always relational those alibis always exists between people and similarly what it decidable surfaces all these interactions those don't exist you know in isolation they exist when those people start coming into contact when we're talking about security now we need to think
about yeah we to go jump back to the loops we talk about what are the ways in which someone might come the harm in this system where are the negative security
outcomes so this is where having something like a scenario flow where you've got you know a set of a set of tasks the user might be performing the context will be performing them and in the performing tasks in and you know what the kind of steps are even in rough sketch form and at a fairly early design phase and this gives you something where you can start saying OK all I can see a spot for an adversary there I can see a spot from anniversary there and this is where you have
to be working with you uses this all has to be co-design because you don't necessarily understand you know what this what this adversaries actually look like what people are worried about in some context you can have better visibility the new users are but a lot of
context you can have worse or you couldn't get teams from very different parts of the system to come together all security at this kind of design level is cross domain and 1 of the things that security people like to spend a lot of time thinking about is 0 this is a physical security problem in this is a network security problem in the digital security
problem there's really no such thing as this level right all security touches all of the different domains all your scenarios probably touch all
the different domains which means you need people from all of those different domains working together so 1 of the things which I like to use to talk about what's a year and yeah what a tool that you might give to a user is an invariant right invariant is something that you're going to ensure doesn't change in the context of the system that you're building on confidentiality is an invariant that people talk about a lot right you know the system will make sure that no 1 other than this set of authorized people can get access to this piece of information under any conditions now the user probably has to do certain things to maintain that invariant which is very useful because you not adding tasks to what the user has to do and there are a lot of different variants in of deniability can be an invariant elicit a complicated 1 for a lotta reasons on a no not on ability you know being able to ensure that any time a user has taken an action they're always going to have to stand by that action if you're designing a financial system this is something you probably care about a lot invariants are things that you can deploy at will and is a lot of complexity there it's 1 of the big things that you can come to enable
users to shape their experience with you users are also 1 of your biggest assets when you're trying to build secure systems or systems that have interesting
participation structures you use use a smart this is something that again especially security people a are often not graded noticing and you users are very good at certain kinds of tasks earlier we talked about stuff that users are bad at I if you want for instance your users if you want something in the system to understand a pattern and to notice when that pattern changes like a pattern of behavior users are gonna be better at that than any chunk of code you can write for the most part of it's a pattern of something that actually affects their life in there and those kinds of daily rhythms and ceremonies as a a a term of art from security which is how what we talk about the steps that someone has to interact and has it take
to maintain an invariant their ceremonies for participation to and a lot of the time they get buried and sometimes we
call parts of them business rules and then the cop call parts of them you why patterns it's very rare that you have a team that actually has a catalog of all of the different ceremonies at the system interacts with and what those ceremonies are intended to do 1 of the things that you would like your system to do if at all possible is to
leave your users smarter than they started using your system and you have an opportunity to interact with users often fairly deeply you have an arbitrary on opportunity to teacher uses things users are never going to come to any system that you build with a full understanding of how to do everything that that system can enable them to do and if they do then you're probably designing a really boring system you should maybe try and build a system that lets them do something interesting on student have to teach research right and because you're going to have to teach them you should probably think about how you're going to structure that teaching and how it interacts with the security properties and the participation properties
you know you need to make certain kinds of things very legible to certain kinds of users who might be good fits for certain kinds of activities on the site I you know for instance 1 of the things that have 1 of things that happens in some contexts and it's a big problem in games you get someone who joins the system you know joins the game is still trying to figure out the way around and what does what and then immediately just cut piled on by a bunch of players like great fresh meat let's take on to resources etc. that is a failure to manage the learning process now in games people are actually a bit better about thinking about this because they do you think about it play they think about performance they think about those kinds of narratives more explicitly and how many people here use Photoshop have used Photoshop how many of you remember the 1st time you open photoshop and it was basically incomprehensible or better yet again of you that is a tool which does not have a very good learning journey you know it's designed for experts and it's expected that a this is kind of the benchmark tool for this industry you're going to spend the time learning you probably to
engage withthem structured tutorial content whatever there's you know the other ways that that learning process is managed but if you don't have any of that it's still a terrible
experience and when your failure or a kind of floundering and clicking around in the system has very serious real world consequences to you and you can't just sort of Explorer this is a much bigger problem if you had the same set of problems learning to use learning to use your bank's website that you have learning to use photoshop and you ended up you know crashing at a bunch of times and that kind of thing that would be significantly more problematic 3 financial and guessing some a very good secure system teachers its users how to be more secure outside of the context of that system it actively enables the kind of learning that stays with those users and makes them better at whatever the tasks that they're trying to do in the world you know without regard to just that system similarly a system that's really good at creating solid participation dynamics builds better communities and and this is true whether or not you're building a social network again every 2 was participatory every tool has these structures if you
build it correctly you end up believing a bunch of extra value on the table I is something called Metcalfe's law which is that the value of the system is proportional to the square of the number of nodes in the system are Metcalfe was the guy who invented Ethernet and this is you
know kind of what he came up with as he was realizing that hey this Internet thing to be worth the money someday I now 1 of the problems in this is a problem that we're seeing right now
it's not actually the square of the number of nodes it sort of the square of the upside value that the system leaves on the table and for each user because if you have a system that tries to capture all of the value right if you have a system that says well build communities but not communities you can take anywhere else they're not going to be
worth it assuming they're not really worth anything else outside of this context that's not actually a very valuable system it's not actually very valuable over useful for for
anyone yeah it so once you've understood it kind of what
you want people to be doing like what are your goals for interaction where your goals for security right if this process flow happens and an adversary tries to step in here this is the set of things that the system should help happen right I this is the set of interaction roles that we think are going to happen these are the places where we think that they're going to be problems this is how we're dealing with those problems this is how a steering people towards interactions that connect they're going to build better communities and this is the thing that you need right this is the thing that the design practice other
than fundamentally what is the what is the system going to be this is the thing that you actually really care about because then this is what
tells you do you know all of the all of the detail of how you build the rest of the system and 1 of the things that we find again and again is the capturing this detail capturing the intuition for 1 of the
interactions that we want what are the interactions that we don't want to how do we shape those and all the kind of fine details of that shaping is 1 of the hardest things especially when you're looking at large design teams large products of it's much easier to do this stuff iteratively
rate and if you have to build a bunch of stuff and then throw it out to the world and actually release it for real you can have a much harder time than if you can basically be playtesting right playtesting is what people do in the game's world and I think it's what people should do in the security world as well right if you have 0 this set of invariants and this interaction with this kind of adversary we think will produce this sort of result you know make a game of that you know make some kind of way of testing it before you've written any code before you've committed to a bunch of requirements stocks get that stuff in their early on you can't
do this without people who look like your users if no 1 on your design team looks like a users you have a problem I don't be apple designing giant phones for a market that's half female and then wondering why people are annoyed and
any time you have you know that the more distance you have between the set of users and the set of people building the system the more work you have to do to ensure that intent crosses that gap and specifically the places where you are likely to run into real problems without intent transfer are in this area are what are the set of adversaries what a set of resources that people have to
draw what are the tactics that are actually going to work for different categories of users there's
really no way to figure that out that kind of thing out without testing so once you have an understanding of what you're what your security goals here and your participation goals are now you can do the rest of the development now of course this is not the giant front-loaded pile of work this is actually you know do a little bit of
design do a little bit of building check it move back and forth you know it's the same process as anything else on however you you know you obviously have to have community in there especially for participation sketchy is fine you don't need to have everything nailed down because you can get it wrong the 1st 5 times anyway and everybody does the and 1 of the things that I see repeatedly causing problems is a failure to capture either the original assumptions or the original community interactions the more of those artifacts you can have to draw on when
somebody in the development team summing the testing something operations is like why are we doing it this way so we're doing it this way because all the way back up to a story from a user about how a thing works in the world the easier you can make that process of chaining through that that set of of artifacts the easier of the time you're going have it for all of these kinds of things when we're talking about the structure of security processes the structure of design processes when you care about is not the process you care about the criteria rate you must be at least this tall to enter the requirements phase you must be at least this told and a development you need to have certain things figured out before you start from different processes and if you don't know what your Design Goals are you have no business writing requirements so that's what I've got free today I'm happy to take questions and I'm happy to jump back into stuff as people have suffered they would hear more details about sold the questions use phrase and on
time 0 yeah quite
be thank you for your lecture in might like to know how you
design and systems that's very can you repeat the question just of letter and how do you design the and of systems the essence of system of such just CIO right the end of systems I so that depends really what the system does you know if you have a system that a
lot of people are dependent on it you have an obligation to the people who use that system to work with them to move out of a system so I mean it's actually really very rare that you have an end of the system you have at end of a technical system but you don't have an end of a set of processes in the world so they're always needs to be some kind of transition plan I now I mean sometimes a given company is in a position to do you to manage the transition plan or something like that but I think it's something and what see this much more as we start looking at like Internet of Things type technologies as we start looking at more and more and infrastructural tools that have software inside them that managing those ends probably becomes a legal responsibility of how many people here so the the Google revolve shut down announcement from the company
that makes nests now at committed that made nest before they got bought by Google on
had this kind of home automation hard that would like you know manager and you're smart furnace talking to something else and this and that and all of your lights talking to each other and Google just decided you know it it's not worth supporting these there aren't that many in the world were just gonna push a firmware update it turns them all off so whatever
infrastructure you had built on top of that it's not just broken and you have no recourse and you have no way of you know blocking it from updating a doing anything like that on if that system you know and probably a few years even
in this case but if that system say was managing something that was critical for people that would be criminal negligence on Google's part here you don't get to just turn stuff off so some kind of transition plan if it's not profitable for 1 company there probably needs to be some kind of hand and really this is the thing that as a society we're going to have to make decisions about but we see things inferences in the medical world where it's like well yeah you're implant is end of life it you know and so we don't have any hardware that can talk to this medical devices anymore are more we do have hardware there's these 3 machines in the world and if any of them break there were no spares and the company that made them is bankruptcy and good luck I so yeah it's it's a challenge that were never again at right now so any more questions we reason but the thank
you thank you for a wonderful talk my my mind has been curriculum and what I don't
quite understand the way you describe the model is and how do you account for all uncertainty if you would not sure you understand the system deeply enough to know where you all white spots are over how do you how a how you approach Brasiliense against things you can't come for you I've been
generally what you do is you try something and then you fail in public and hopefully no 1 gets killed and then you try something else on and all of
this stuff is always but 1 of the fundamental things and and this guy really drummed into me in high risk world the place the the place you have to start when you're doing the
especially security work and also even and participation work is that you're going to fail right some people are going to be bored by your system are going to leave some people are going to get compromise are going to get hurt this is unavoidable because we can't build perfect systems in the 1st right we probably can build perfect systems ever in many cases on but you do it you can write ends I think this is 1 of the things we're having users involved makes a big difference because they are going to give you a much better read on weld yeah that tactic might work or it might not work I universe is deniability they mentioned earlier is a property that some of the designers of secure messaging systems are really fond of and when they point to you when when they're asked why they point to like times and I've talked to lawyers and lawyers said you know defense lawyers have civil you assure you know claiming that my clanking cryptographically deny a message might be worth trying and caught on but what they don't realize is that's not 0 yeah this will work that's well yeah sure try anything I mean whatever might work in-house I get look at some point but and to capturing like that understanding of like is this is likely to work is is unlikely to work is this reasonable etc. is a core part of that kind of a the elicitation of the adversarial dynamics at the OK I think it is normal
and there is no I think that's it's so thank you very much it mn many ahead