Don't get Distracted

0 views

Formal Metadata

Title
Don't get Distracted
Alternative Title
Finding Responsibility
Title of Series
Number of Parts
69
Author
Thompson, Caleb
License
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Identifiers
Publisher
Confreaks, LLC
Release Date
2017
Language
English

Content Metadata

Subject Area
Abstract
In 2011, with a team of interns at a Department of Defense contractor, I created a Wi-Fi geolocation app to locate hotspots. It could find the location in 3D space of every hotspot near you in seconds. We made formulas to model signal strength and probable distances. We used machine learning to optimize completion time and accuracy. I was so caught up in the details that it took me months to see it would be used to kill people. What do we do when we discover that we're building something immoral or unethical? How can we think through the uses of our software to avoid this problem entirely?
Loading...
Medical imaging Content (media) Torque Content (media) Dependent and independent variables XML Descriptive statistics
Android (robot) Computer program Randomization Group action Source code Range (statistics) Client (computing) Disk read-and-write head Perspective (visual) Information technology consulting Dimensional analysis Computer configuration Velocity Code Cloning Office suite Error message Amenable group Mapping Software developer Electronic mailing list Sound effect Prediction Computer Flow separation Digital rights management Process (computing) Hash function Quicksort Mathematical optimization Reading (process) Point (geometry) Slide rule Computer file Student's t-test Mass Number Goodness of fit Latent heat Selectivity (electronic) Router (computing) Address space Form (programming) Pairwise comparison Graph (mathematics) Information Key (cryptography) Physical law Set (mathematics) Line (geometry) Extreme programming Convolution Calculation Uniform resource locator Loop (music) Software Video game Wireless LAN State observer Building Length Code State of matter Multiplication sign Direction (geometry) Decision theory 1 (number) Combinational logic Design by contract Mereology Computer font Likelihood function Optical disc drive Mathematics Matrix (mathematics) Bit rate Linker (computing) Software framework Series (mathematics) Position operator Area Covering space Curve Algorithm Email Electric generator Software engineering Process (computing) Rational number Fitness function Variable (mathematics) Functional (mathematics) Measurement Connected space Degree (graph theory) Data mining Computer science Self-organization Normal (geometry) Website Data type Resultant Normal distribution Sinc function Filter <Stochastik> Trail Game controller Service (economics) Clique-width Identifiability Great circle Virtual machine Distance Theory 2 (number) Wave packet Root Integer Software testing Task (computing) Addition Multiplication Polygon Projective plane Estimator Blog Vertex (graph theory) Local ring Separation axiom
Computer program Group action Building Hoax Transportation theory (mathematics) Code State of matter Euler angles Multiplication sign Decision theory Source code Shape (magazine) Disk read-and-write head Explosion Facebook Casting (performing arts) Bit rate Hypermedia Core dump Negative number Videoconferencing Cuboid Office suite Physical system Thumbnail Email Touchscreen Process (computing) Decision theory Software developer Electronic mailing list Sound effect Bit Term (mathematics) Computer Demoscene Hand fan Arithmetic mean Process (computing) Hausdorff dimension Website Right angle Thermal conductivity Implementation Mobile app Service (economics) Divisor Virtual machine Vector potential Code Event horizon Automatic differentiation Open set Number Twitter Revision control Device driver Term (mathematics) Operator (mathematics) Software Energy level Implementation Hydraulic jump Form (programming) Operations research Default (computer science) Key (cryptography) Projective plane Physical law Code Planning Line (geometry) Group action Local Group Newsletter Word Uniform resource locator Software Logic Personal digital assistant Device driver American Physical Society Statement (computer science) Dependent and independent variables Speech synthesis Object (grammar) Family Local ring
Point (geometry) Code Multiplication sign Decision theory Online help Mereology Food energy Power (physics) Goodness of fit Database Core dump Best, worst and average case Subtraction Traffic reporting Address space Physical system Task (computing) Email Information Forcing (mathematics) Software developer Projective plane Electronic mailing list Fitness function Digital rights management Process (computing) Software Personal digital assistant Password Website Self-organization Figurate number
Computer
the the the
the the the the this talk is called
don't get distracted He's the called finding responsibility but this name work that he should be a little after my brief introduction and hit the ground
running with some potentially disturbing content includes references to but not descriptions of killing in society we stay the next 2 minutes decide on another torque if you'd like to go all the judgment images in in the middle people we have been given you yes
the slides a blank no they're not broken I've been a developer for about 10 years ever batches degree in software engineering work and suffer jobs in multiple industries including a consultancy where I work with many clients very Eitel tools from a social support network for people with type 1 maybe these to shipping may comparison service I even work for banks building task management software now I work for a directive on supporting where I get help also to developers like you mind your code and solve your interesting problems on our for the past 4 years have organized keeper be we're a community-oriented ruby Conference in Austin Texas I'm going to tell you about how I took a job building software to kill people but don't get distracted by that time I did not have time even before I walked across the stage a graduation I accepted an offer for an integer it paid half as much as the most ever got my highest paying job not to mention that I just spent 4 years working low-paid student jobs are living just on student ones I'd be doing a contract company for the Department of Defense the Department of Defense or deity is the part of the government made up by the military in it's the deity outsources all sorts of things from PATA multi-mission maritime aircraft to blue shade 451 poly will they also tears a lot of software projects at the time I thought nothing of the fact that it be helping military besides dividend as right my dad was in the military so as my grandfather I have great respect for those serving and tears is good money as a great opportunity and a good friend of mine that they get life was good I should have for my 1st they work in North Virginia or no the industry likes to call and that the team of other interns and I learned about what we will be built a tool to build a tool to find Wi-Fi signals using your from it seems pretty cool compared what I build up to that point the most complicated thing I'd ever done was an inventory management service the entity concerned solve over much with persistence who really is a stop and restart a program and that it was right there in memory and if a forgot how many errors CDs you had that cares honestly the idea of finding Wi-Fi routers based on the signal strength seemed pretty intimidating the idea impressed but don't get distracted by other the software was intended to kill people I did the team effort to get started on the project the gist of the tool was that they would look at how Wi-Fi signal strength changed as a form of ground if the signal strength got stronger you're probably moving closer to the source if a guy weaker you're probably moving away define this information we collect 2 pieces of information fur each Wi-Fi signal and range your phone's location and the signal strength of that white rice to predict the actual location of the Wi-Fi signal we used a convolution of 2 other the 1st was square and measured the distance between the signal strength we'd observed and the expected signal strength at that distance and I did that for every point in the search graph locations they the lowest R squared error rate were the most likely to be the source of the Wi-Fi we combine that calculation with Agasi an estimation it creates a probability curve using the standard distribution or the bell curve the probably seen all of us it started with an inverted curve a probability hole of how low likelihoods that the Wi-Fi signal originated from those distances that represented the idea that the phone is probably not standing right next what you're trying to find it added in normal bell curve there is a high probability that the source originated from a distance further out the album adjusted the width and height of each of those curves by consulting past migrants it created a heat map of probabilities for the sciences we normalize those two probabilities for each location and search grid and we would combine the combination of those 2 organs with much more correct than either individual we start this probability matrix for each location that font collected from using these we're able to give you the distance if you moved in a straight line for how far away or from a Wi-Fi source if you turn a corner corner we can also tell you directions lead find it in 2 dimensional space clients users will the altitude as well the technology was the most interesting thing I built to that point it may still be the most interesting thing I built but don't let that distract you it was designed to kill people I mentioned earlier that I had a software engineering degree at this point my teammates were much earlier in the educational careers most of them were a year to and for your programs in more mass for computer science majors my expertise was in the design and process of building software while there's was in the theory of mathematics for how computers are used and help to translate the working novels that they built in MATLAB into the job because that would need to run on Android phones and let's be honest with him plenty of time deciding whether we prefer eclipse on that these can I say how happy and that is a ruby developer had not had to figure out where to put a jar file in almost 5 years this is a lot of time bike-sharing encode organizations and pairing on performance improvements to it worked but it took almost 7 minutes to find the Wi-Fi sources this is partially because the code we translated from Matlab which is optimized for working in matrixes of numbers we wrote in nested loops quadrupling nested loops was some account of calculations the 1 example is that we were calculating the distance between 2 points we used great circle distance which is where you measure the shortest distance between 2 points on a sphere such as the function performing the calculation was being hit hundreds of thousands of times for each collection points often with the same 2 locations it was very slow we solve that by implementing a hash with the key is where the 2 locations were of the 2 locations and the values of the distances between this at least not that we don't have to redo those calculations that another optimizations we made sped up the performance from 7 minutes to a few seconds but don't be distracted the performance made faster to kill people the accuracy of the locations wasn't fantastic I don't remember exactly what it was but the number 45 feet sticks in my head that's about the length of the shipping container that significant when the Wi-Fi signal strength of 8 2 11 and is only about 100 feet that meant that we could be almost half the range of that Wi-Fi router away from where I was 2 big area I talk about the Garcia estimation the 2 curves that were part of the 2nd of we hard-coded the numbers that defines the width and depth of curves there were only starting points but they were starting points that we used every time we make this calculation it is a new node a genetic governance it's a type of machine learning program that produces a set of values that optimize for a desired result each of the Gauss ian estimation values is a gene the set of values in a gene known the set of values is a gene known in genetic over the genes were 3 counts a big assumption is what the genetic cover the users to prefer to measure the performance of a gene for my data sets of readings I was using I knew the actual location of a Wi-Fi source so I was able to plug that gene into the geo-location algorithm and measure the distance between the location that it found in the location that I knew was that that was my fitness for the smaller the better genetic algorithms take a set of genes called a generation and they keep a certain percentage of pop performers these performers survive to the next generation as clones sometimes the over the mutates those clowns by adding a Garcia and random value to each G that meant that each of them had a slightly a slight
chance of performing better or worse new random genomes were created for the remainder of population we said the top performers across all populations which meant that when I finished the genetic or I could look at the best performer and plug that into our geo-location of remember others geo-location under the run over the weekend he was able to increase the accuracy from 40-odd feet to about 10 comma decimal 3 25 per cent the year of the original that's less than the GPS accuracy on the forms that we were collecting data from which means that it was probably not accurate overall it's called overfitting and in it solves by using separate sets of test and training data I love this genetic algorithms are squared Gustine random values it's a sort of thing you hear about and you learn in school but they tell you you'll never use again after you graduate but we were using it in a real project it was great but what that destructive this made it easier to help people the location of a network accurately and quickly the next features was and tracking of moving Wi-Fi access point I briefly wondered where Wi-Fi access point would be but that wasn't as interesting as figuring out how to find it with our we made use of common filters to this to observed 3 state variables the position velocity and acceleration of a Wi-Fi signal source given these values in the time since last measurement of common filters able to improve the current prediction surprisingly well and it throws away values that have low accuracy automatically each time we ran the real-time our then we also run the common folk with only that information was able to produce an estimate that was more accurate than the calculated value and that showed movement at the same time we have the ability to track more than 1 wife I said we felt collection of readings by the unique identifier of each signal known as the MAC address the filter datasets went through the full algorithm to produce predictions we use the at the eyes of funds to pull the location data for each Wi-Fi signal on sort in range basically anything you can see in your Wi-Fi connection list it was all very exciting the seem like academic problems but we were getting use them in a real-world project being a programmer was going to be great but don't let that this meant that we were able to kill multiple people suffer we've been working with the project on throughout this process this fairly laws a fair about most things he'd check and maybe once a day and then go back to the part of building dedicated to undercover to classified work whatever we 1 of these milestones we'd he'd be happy about it but it question always came up he wanted to sniff out signals but up iPhones in addition of those without my Wi-Fi too much harder problem from a technical perspective the functionality necessary to do this is called promiscuous it's a sitting on the wireless network control neither Android nor iPhone support this option so we have to either root or jailbreak the funds we looked for packages that would help a sniff out this project this these packets as they were sent back to the router we found something on SourceForge but it was a well documented and we didn't understand it very well we told the project we get back to this later notice that was that important we had this Cook Technology defined use Wi-Fi sources and that was the goal it time we demonstrate a new and exciting tech the same question came up we got Wi-Fi signals located does a fine fonts is seeking seconds instead of minutes great designed fonts we looked in defining phones and it doesn't seem likely but I will get back to later but wouldn't come back to later we got moving targets working defined front I had been distracted all the cool problems were solved finding nodes speeding things up making more accurate predictions is also much so cool so much fun I hadn't about why we were putting all this work and finding a better place a sitting get good wife that doesn't make sense if you think about it for 1 of his eyes does it find funds this was never about finding a better life he was always about finding from phones carried by people remember I said I work for department of defense contractor the deity is the military I was building a tool for the military to find people based on where the funds were and until a try rationalize the military's and placed effect truth justice and the American 1 but this is the same time that we found out that the government had been spying on us in the United States with and that it went out that technology told local state and federal gov government officials law-enforcing officials over 700 times to run their own actions the military is the tool of the government and it seemed like we couldn't trust the government as much as we thought we had I didn't wanna be a part of something that was going to use to kill people especially since I never know who is used against let alone have a say in decisions rationalism to we were in chance we didn't have a clearance the projects and this company did for the government were top secret I was allowed to know what they were I could probably get shelved and forgotten about this is an extreme of an example of code that was used in a way that the developer did not and intend the project owner conveniently left out its purpose when he was explaining the Goals and I continue we didn't look too hard and was great pay for me at the time it was a great project maybe I just didn't wanna know what would be used for I let myself get distracted I was distracted by the technology but would be just as easy to be distracted by cool framework that the company with you the great design of an apple looks super good on your portfolio or some really nice office amenities maybe others are doing the same thing and clearly they've already thought about it so must be fine there are other examples of when code was used in ways that it was wasn't and and of code the just as bad things a year and a day ago a developer named Bill Sauer wrote a blog post it opened with the line if you write code for a living there's a chance that at some point someone will ask you to do good something a little deceitful if not outright unethical Bill had been asked to create a quiz that will almost always give a result the benefit of the client FIL worked in Canada in in Canada there are laws in place that Lennart how pharmaceutical companies can advertise prescription drugs patients anyone can learn about the general symptoms addressed by drug but only those with the prescription thing it's specific information because of this law the quizzes posing as a general information site rather than an advertisement for a specific drug if the user didn't answer that I did they were allergic to the drug or that they were already taking it every results and ask your doctor about this drug through the requirements of do and that's what bill could at the project manager did a quick check and told the litter didn't seem to be working they always give the same answer those rural requirement said about a little while later though that an e-mail from a colleague it contained a link to a news article a young woman had taken the drug that Bill had written this quest for she killed herself it turns out that some of the side effects of this drug odd severe depression and suicidal thoughts nothing build did was a legal like me FIL as a young developer making great money and doing when you the purpose of the site was the push a specific project That's why I was being built FIL in charter select marketing he never intended for this to happen maybe Bill got distracted his conclusion bill as
developers were 1 of the last lines of defense against potentially dangerous and unethical practices we are approaching a time with the software we build will drive vehicle that transports your family soccer practice there already AI programs that help doctors diagnose disease it's not hard to imagine them recommending prescription drugs and to more software continues to take over every aspect of our lives the more important it is for us to take a stand insure the ethics are ever present America since the day I was trying to think twice about the effects of Michael before writing I hope that you will too I think it's poignant that all of the examples that the listed as something that might happen in the future happened today FIL story isn't that far off from but they're still other examples earlier this year and the story came out about Cuba it built into its ride-sharing apps code that was called gray ball It's a feature of but their of their violation violation terms of service tool that can populate the screen with fake cars when it's Apple's opened by users in terms of in violation of its terms in a statement who were said this program denies ride requests to users who were violating our terms of service whether that's people who is Sikh that physically drivers competitors looking to disrupt our operations or opponents who collude with officials on secret stains meant to entrap drivers in practice is the New York Times reports it was used in Portland to avoid code enforcement officers looking to build a case against Uber for operating without a license when triggered by Members logic it populates the app with cars that don't exist with fake drivers to quickly cancel after accepting a right I'm not a lawyer but this seems like an obstruction of justice itself a crime outside of the illegal activities in grave always use even today though mostly outside the United States I'm a huge fan of rage services if thought uncommon seen the news these days articles about these driver doing heinous things Grabel may have enabled some of those things that again this is an unintended consequence of a tool that was built maybe internal pitch for this was a gray ball users who were violating the terms people were under 18 people who didn't pay to clean up the explosive accident last night rather than block them potentially cousin to create a new Account they could just be put into an alternate dimension where they could never get a ride for some reason that's fine if these developers had thought about the worst possible use for this could the circumvention of justice might have come up earlier on it could have been addressed maybe they were distracted by the face value of the request for looking deeper its purpose there are also 2 things as well aren't as black and white APS that listen to the microphone to tailor ads to you based on what you saying here from websites designed to exploit psychology to take up as much of your free time as possible which of them linked to its loading rates of depression any number of apps about you into an e-mail newsletter this ad is obviously bad as the previous examples but at least in my opinion there still kind of shape this value system is different from ours richer Solin for example believes that e-books around the others may think that's a little eccentric but that viewpoint agrees with his overall system of elites they're actually words for what society decides a good or bad versus what you Arieh Richard Solin individually ethics and morals all modern psychology or uses these terms interchangeably and understanding between you and I will be useful later on ethics from pose when outside groups science society of profession a community like ours or where you live religions provide ethical systems so 2 groups of friends societies in whatever form determine right or wrong good and bad and they impose those definitions on its members ethics in societies and the local state and national levels are often coded in the law morals on the other hand more personal version of the same society as a whole and poses it more as on smaller communities and all of that trickles down to the individual level that's not to say that your morals can conflict with societies at maybe you believe that freedom of speech is a core tenet of human rights but you live somewhere where 2 facing political or religious objects is wrong was soccer distracted by morals and ethics just yet though will come back to those later the unifying factor of all of these stories is that developers built codes that did these unethical or immoral thing as a profession we have a superpower we can make computers do that's we build tools and ultimately some responsibility lies with us to think through how they're going to be used much of what their intentions what misuses might come out of None of us want to be building things that will be used for evil the Association of Computing Machinery is a society dedicated to advancing computing as a science and profession ACM includes this in the code of ethics and professional conduct well intended actions including those that accomplish assigned duties may lead to harm unexpectedly in such an event the responsible person or persons per obligated to undo or mitigate the negative consequences as much as possible 1 way to avoid unintentional harm is to carefully consider potential impacts on all those affected by decisions made during design and implementation so how can we carefully consider potential impacts honestly I don't have the answers I don't think that there is a universal into because I have to believe that if there was we wouldn't have this problem we wouldn't billion building this code in the 1st I do have a couple ideas 1 I got from my French means add to the planning process the step where we come up with the worst possible use for our software opting in folks to a mailing list by default the worst case is probably that we send them a bunch of e-mails and the unsubscribe maybe they stop being a cast machines said I willing this so my hypothetical start-up soul for a beer mailing list especially when that might be all the key for company afloat when I and I can see that I understand it I still think it a little bit shading is not a best practice that is not physically hurting anyone if I had sat down and thought through what my code would be used for when I was building the Wi-Fi location on would come to a much different conclusion actually I think that thinking through the worst possible uses of the code can be a fun it's a fun exercise may come up with some pretty lacking examples if we send back on an e-mail and has notifications for new e-mails on his thumb that he might be looking at his iPhone when the Riddler drives by in the river and he might get off his woody one-liner the crime scene really this really that Who's afraid of the big black bat It's also plausible but it shows that you can come up with some pretty wild out there there are obvious at 1st glance another thing that I think I should have done that we can all do more of is to just not take the request at face value the project under the defense contractor didn't spell out what the code will be used for but at least in retrospect it was a big jump in logic we're going to build an app find Wi-Fi signals is all true but it's not the whole truth asking myself where them why enough times might have led me to an earlier understand White define the Wi-Fi signal source why to go to that why why comedian Kimelman johnny best known for the TV show silicon valley in his recent film The big 6 that the footer recently on the subject I know there's a lot of
scary stuff in the world right now but this is something I've been thinking about that I can get out of my head as a cast member on a show about tech job entails visiting tech companies conferences etc. we you both eager to show off the often see stuff that's scary I mean weapons I mean altering videos stuff was off obvious ethical issues it will bring up or concerns them were realizing that 0 consideration seems to be given the ethical implications of that I didn't even have a paper the they're shocked at being passed which means that nobody is asking these questions they were not making it for that reason but if people choose to use it that way it isn't our fault safeguards will develop but tech is moving so fast there's no way humanity or laws can keep up we don't even know how to deal with open death threats online only can we do this never should with us we seen that seemed blase attitude and how Twitter and Facebook deal with abuse and Tech has the capability to destroy us we see the negative effect of social media no ethical considerations are going into the development of these technologies you can put this stuff back in the box once it's out there it's about that and there are no guardians it's terrifying it's a
major problem we were given so much power but we can't do anything to ensure that its use it safely thinking about what we're doing and being careful not to build things that can be used maliciously is really important and police that we can be this is chosen
in an interview with The New Yorker radio our she's discussing the ethics of what developers which is not I guess technologist should realize that we have an ethical obligation to make decisions that go beyond just meeting deadlines and creating a project but actually take some chunks of time and say what are the consequences of the system how can this be used how could be misused the starter figure we can mitigate a software system from being misused or decide whether we want implemented at all asked do there are systems that if misused can be very dangerous don't get distracted but deadlines and feature requests think about the consequences of what you're building to build in safeguards to prevent misuse or don't build at all because it's too dangerous I'm asking you to do something about it well I guess I'm asking you not to do something because of it's only fair that we talk about when and where to take that step let's I had attention and I can go back and 90 2011 and do it all over again the tool is just to me the 1st time what do I say I think the 1st thing is to establish a mutual understanding of the task it's entirely possible this 1 that I don't understand what the actual thing it's and I'm overreacting I ask why are we finding these people the project under says we wanna find people cell phone who is finding them and why I ask Anna probably some soldiers in the Middle East why the repeat I can't tell you that I can't tell you that is something that I get a lot from project it's good for I have a Clarence and I know things about this project I know what you're asking and I know the answers but I'm not allowed to tell you at this point I think we have a mutual understanding the task is to help soldiers find people's thoughts probably attached to those people the reason is left unsaid but I think we both know that this organization is a defense contractor they build things for the military that is their core competency they're not not going to do on the other hand I care a lot about not killing people the company's goal is to kill to to build things so the military the military is necessary and sometimes they need to use force personally however I don't wanna be part of that if my goal is not to build those tools and there is a good fit for me at this company that probably means that the worst case here is that I'm the of today without a job either I'll say no and they'll fire me for all saying in I'm not comfortable with this but good luck not that those a worst-case scenario they're not necessarily what's going to happen and so before I do this I need to consider some questions can I afford to leave here without this job financially Caroline my personal network to give me a new job have I built up enough trust of my employer where I can go to this and be heard out the answer for me at the time was now at all of those questions sometimes something is important enough that you still need to do it but that's a very personal decision there's a lot to go into these decisions and they have consequences I would like to think that I would still say let's look at another situation where someone did the ethical a developer will call Alice received a strange request we want to identify weak passwords in our system and notify users to change we'd like to run a password cracking tool on a very large database it House thought this was kind of a strange request but she said that if the appropriate paperwork were filed she would I received the paperwork and she ran a crack the next request was would like they list of users e-mail addresses as well as their passwords Alice knew that her co-workers had a valid desire to tell customers improve their passwords she also knew that a lot of users reuse credentials across websites if a report was created the combined those 2 pieces of information it could be misused to log in to another to those users accounts on other websites else pointed the center manager and together with the customer success team they designed an e-mail that I didn't include the password customers receive notifications about the weak passwords and were able to change nobody got fired and I was built up trust with energy different scenarios required different ways of thinking about what should do sometimes the right thing to do is to say nothing and just do the work it is a simple fine and that has consequences but don't get distracted by having to think about it sometimes your code can help people the the this
and
Loading...
Feedback

Timings

  667 ms - page object

Version

AV-Portal 3.10.1 (444c3c2f7be8b8a4b766f225e37189cd309f0d7f)
hidden